2023-04-28 08:53:32,530 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3 2023-04-28 08:53:32,543 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-28 08:53:32,560 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-28 08:53:32,560 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/cluster_6953ce0a-bffa-891d-c013-266678f3cf8c, deleteOnExit=true 2023-04-28 08:53:32,561 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-28 08:53:32,561 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/test.cache.data in system properties and HBase conf 2023-04-28 08:53:32,562 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/hadoop.tmp.dir in system properties and HBase conf 2023-04-28 08:53:32,562 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/hadoop.log.dir in system properties and HBase conf 2023-04-28 08:53:32,563 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-28 08:53:32,564 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-28 08:53:32,564 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-28 08:53:32,688 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-28 08:53:33,125 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-28 08:53:33,131 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-28 08:53:33,132 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-28 08:53:33,132 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-28 08:53:33,133 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-28 08:53:33,133 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-28 08:53:33,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-28 08:53:33,134 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-28 08:53:33,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-28 08:53:33,135 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-28 08:53:33,136 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/nfs.dump.dir in system properties and HBase conf 2023-04-28 08:53:33,136 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/java.io.tmpdir in system properties and HBase conf 2023-04-28 08:53:33,137 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-28 08:53:33,137 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-28 08:53:33,137 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-28 08:53:33,718 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-28 08:53:33,723 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-28 08:53:34,054 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-28 08:53:34,242 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-28 08:53:34,267 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-28 08:53:34,321 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-28 08:53:34,355 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/java.io.tmpdir/Jetty_localhost_40057_hdfs____w5imbk/webapp 2023-04-28 08:53:34,507 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40057 2023-04-28 08:53:34,572 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-28 08:53:34,573 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-28 08:53:35,049 WARN [Listener at localhost/33889] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-28 08:53:35,211 WARN [Listener at localhost/33889] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-28 08:53:35,236 WARN [Listener at localhost/33889] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-28 08:53:35,244 INFO [Listener at localhost/33889] log.Slf4jLog(67): jetty-6.1.26 2023-04-28 08:53:35,251 INFO [Listener at localhost/33889] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/java.io.tmpdir/Jetty_localhost_45095_datanode____dh2d1u/webapp 2023-04-28 08:53:35,382 INFO [Listener at localhost/33889] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45095 2023-04-28 08:53:35,735 WARN [Listener at localhost/39717] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-28 08:53:36,270 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x83e241becb15c79e: Processing first storage report for DS-2cf46bf0-3876-4701-bac8-b95080c59f01 from datanode 300a562e-94bc-4fc2-8ca0-a0cb56059f0c 2023-04-28 08:53:36,272 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x83e241becb15c79e: from storage DS-2cf46bf0-3876-4701-bac8-b95080c59f01 node DatanodeRegistration(127.0.0.1:46297, datanodeUuid=300a562e-94bc-4fc2-8ca0-a0cb56059f0c, infoPort=40535, infoSecurePort=0, ipcPort=39717, storageInfo=lv=-57;cid=testClusterID;nsid=23959741;c=1682672013798), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-28 08:53:36,272 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x83e241becb15c79e: Processing first storage report for DS-52e9b44b-7b2d-4e09-bc24-d49a897d8abf from datanode 300a562e-94bc-4fc2-8ca0-a0cb56059f0c 2023-04-28 08:53:36,272 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x83e241becb15c79e: from storage DS-52e9b44b-7b2d-4e09-bc24-d49a897d8abf node DatanodeRegistration(127.0.0.1:46297, datanodeUuid=300a562e-94bc-4fc2-8ca0-a0cb56059f0c, infoPort=40535, infoSecurePort=0, ipcPort=39717, storageInfo=lv=-57;cid=testClusterID;nsid=23959741;c=1682672013798), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-28 08:53:36,352 DEBUG [Listener at localhost/39717] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3 2023-04-28 08:53:36,439 INFO [Listener at localhost/39717] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/cluster_6953ce0a-bffa-891d-c013-266678f3cf8c/zookeeper_0, clientPort=54655, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/cluster_6953ce0a-bffa-891d-c013-266678f3cf8c/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/cluster_6953ce0a-bffa-891d-c013-266678f3cf8c/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-28 08:53:36,453 INFO [Listener at localhost/39717] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54655 2023-04-28 08:53:36,461 INFO [Listener at localhost/39717] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 08:53:36,463 INFO [Listener at localhost/39717] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 08:53:37,129 INFO [Listener at localhost/39717] util.FSUtils(471): Created version file at hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143 with version=8 2023-04-28 08:53:37,130 INFO [Listener at localhost/39717] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/hbase-staging 2023-04-28 08:53:37,453 INFO [Listener at localhost/39717] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-28 08:53:37,915 INFO [Listener at localhost/39717] client.ConnectionUtils(127): master/jenkins-hbase6:0 server-side Connection retries=45 2023-04-28 08:53:37,947 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 08:53:37,947 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-28 08:53:37,948 INFO [Listener at localhost/39717] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-28 08:53:37,948 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 08:53:37,948 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-28 08:53:38,134 INFO [Listener at localhost/39717] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-28 08:53:38,223 DEBUG [Listener at localhost/39717] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-28 08:53:38,318 INFO [Listener at localhost/39717] ipc.NettyRpcServer(120): Bind to /172.31.2.20:32801 2023-04-28 08:53:38,328 INFO [Listener at localhost/39717] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 08:53:38,330 INFO [Listener at localhost/39717] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 08:53:38,351 INFO [Listener at localhost/39717] zookeeper.RecoverableZooKeeper(93): Process identifier=master:32801 connecting to ZooKeeper ensemble=127.0.0.1:54655 2023-04-28 08:53:38,394 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:328010x0, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-28 08:53:38,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:32801-0x100f6e3c0280000 connected 2023-04-28 08:53:38,425 DEBUG [Listener at localhost/39717] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 08:53:38,426 DEBUG [Listener at localhost/39717] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 08:53:38,429 DEBUG [Listener at localhost/39717] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-28 08:53:38,437 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=32801 2023-04-28 08:53:38,437 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=32801 2023-04-28 08:53:38,438 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=32801 2023-04-28 08:53:38,438 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=32801 2023-04-28 08:53:38,439 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=32801 2023-04-28 08:53:38,445 INFO [Listener at localhost/39717] master.HMaster(444): hbase.rootdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143, hbase.cluster.distributed=false 2023-04-28 08:53:38,517 INFO [Listener at localhost/39717] client.ConnectionUtils(127): regionserver/jenkins-hbase6:0 server-side Connection retries=45 2023-04-28 08:53:38,518 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 08:53:38,518 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-28 08:53:38,518 INFO [Listener at localhost/39717] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-28 08:53:38,518 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-28 08:53:38,518 INFO [Listener at localhost/39717] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-28 08:53:38,523 INFO [Listener at localhost/39717] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-28 08:53:38,526 INFO [Listener at localhost/39717] ipc.NettyRpcServer(120): Bind to /172.31.2.20:40909 2023-04-28 08:53:38,529 INFO [Listener at localhost/39717] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-28 08:53:38,535 DEBUG [Listener at localhost/39717] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-28 08:53:38,536 INFO [Listener at localhost/39717] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 08:53:38,539 INFO [Listener at localhost/39717] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 08:53:38,541 INFO [Listener at localhost/39717] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:40909 connecting to ZooKeeper ensemble=127.0.0.1:54655 2023-04-28 08:53:38,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:409090x0, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-28 08:53:38,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:40909-0x100f6e3c0280001 connected 2023-04-28 08:53:38,546 DEBUG [Listener at localhost/39717] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 08:53:38,551 DEBUG [Listener at localhost/39717] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 08:53:38,552 DEBUG [Listener at localhost/39717] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-28 08:53:38,555 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40909 2023-04-28 08:53:38,555 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40909 2023-04-28 08:53:38,556 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40909 2023-04-28 08:53:38,559 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40909 2023-04-28 08:53:38,559 DEBUG [Listener at localhost/39717] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40909 2023-04-28 08:53:38,562 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:38,575 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-28 08:53:38,576 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:38,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-28 08:53:38,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-28 08:53:38,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:38,602 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-28 08:53:38,604 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase6.apache.org,32801,1682672017275 from backup master directory 2023-04-28 08:53:38,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-28 08:53:38,607 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:38,607 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-28 08:53:38,608 WARN [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-28 08:53:38,608 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:38,611 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-28 08:53:38,613 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-28 08:53:38,699 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/hbase.id with ID: 790b8119-6d87-4e02-a325-574de8bc8ce3 2023-04-28 08:53:38,746 INFO [master/jenkins-hbase6:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-28 08:53:38,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:38,807 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6a3f12c0 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:38,841 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@381161b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:38,870 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:53:38,872 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-28 08:53:38,891 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-28 08:53:38,891 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-28 08:53:38,893 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:53:38,897 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:53:38,899 INFO [master/jenkins-hbase6:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 08:53:38,937 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store-tmp 2023-04-28 08:53:38,970 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:38,970 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-28 08:53:38,971 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:53:38,971 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:53:38,971 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-28 08:53:38,971 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:53:38,971 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:53:38,971 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 08:53:38,973 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/WALs/jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:38,998 INFO [master/jenkins-hbase6:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C32801%2C1682672017275, suffix=, logDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/WALs/jenkins-hbase6.apache.org,32801,1682672017275, archiveDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/oldWALs, maxLogs=10 2023-04-28 08:53:39,068 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46297,DS-2cf46bf0-3876-4701-bac8-b95080c59f01,DISK] 2023-04-28 08:53:39,077 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:53:39,144 INFO [master/jenkins-hbase6:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/WALs/jenkins-hbase6.apache.org,32801,1682672017275/jenkins-hbase6.apache.org%2C32801%2C1682672017275.1682672019011 2023-04-28 08:53:39,145 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46297,DS-2cf46bf0-3876-4701-bac8-b95080c59f01,DISK]] 2023-04-28 08:53:39,145 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:53:39,146 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:39,150 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 08:53:39,151 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 08:53:39,214 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-28 08:53:39,221 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-28 08:53:39,246 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-28 08:53:39,259 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:39,266 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-28 08:53:39,268 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-28 08:53:39,289 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-28 08:53:39,295 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:53:39,296 INFO [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61932200, jitterRate=-0.07713830471038818}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-28 08:53:39,296 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 08:53:39,298 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-28 08:53:39,325 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-28 08:53:39,325 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-28 08:53:39,328 INFO [master/jenkins-hbase6:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-28 08:53:39,330 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-28 08:53:39,368 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 37 msec 2023-04-28 08:53:39,369 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-28 08:53:39,397 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-28 08:53:39,403 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-28 08:53:39,431 INFO [master/jenkins-hbase6:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-28 08:53:39,435 INFO [master/jenkins-hbase6:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-28 08:53:39,438 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-28 08:53:39,443 INFO [master/jenkins-hbase6:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-28 08:53:39,447 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-28 08:53:39,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:39,453 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-28 08:53:39,453 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-28 08:53:39,467 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-28 08:53:39,472 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-28 08:53:39,472 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-28 08:53:39,473 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:39,473 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase6.apache.org,32801,1682672017275, sessionid=0x100f6e3c0280000, setting cluster-up flag (Was=false) 2023-04-28 08:53:39,491 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:39,502 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-28 08:53:39,504 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:39,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:39,517 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-28 08:53:39,518 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:39,521 WARN [master/jenkins-hbase6:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.hbase-snapshot/.tmp 2023-04-28 08:53:39,564 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(952): ClusterId : 790b8119-6d87-4e02-a325-574de8bc8ce3 2023-04-28 08:53:39,570 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-28 08:53:39,577 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-28 08:53:39,577 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-28 08:53:39,581 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-28 08:53:39,582 DEBUG [RS:0;jenkins-hbase6:40909] zookeeper.ReadOnlyZKClient(139): Connect 0x0e388fd4 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:39,593 DEBUG [RS:0;jenkins-hbase6:40909] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ff7063, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:39,594 DEBUG [RS:0;jenkins-hbase6:40909] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c0e4a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase6.apache.org/172.31.2.20:0 2023-04-28 08:53:39,629 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase6:40909 2023-04-28 08:53:39,634 INFO [RS:0;jenkins-hbase6:40909] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-28 08:53:39,635 INFO [RS:0;jenkins-hbase6:40909] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-28 08:53:39,635 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1023): About to register with Master. 2023-04-28 08:53:39,638 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase6.apache.org,32801,1682672017275 with isa=jenkins-hbase6.apache.org/172.31.2.20:40909, startcode=1682672018517 2023-04-28 08:53:39,661 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-28 08:53:39,661 DEBUG [RS:0;jenkins-hbase6:40909] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-28 08:53:39,672 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-04-28 08:53:39,672 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-04-28 08:53:39,672 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-04-28 08:53:39,672 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=5, maxPoolSize=5 2023-04-28 08:53:39,673 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase6:0, corePoolSize=10, maxPoolSize=10 2023-04-28 08:53:39,673 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,673 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=2, maxPoolSize=2 2023-04-28 08:53:39,673 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,676 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682672049676 2023-04-28 08:53:39,678 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-28 08:53:39,681 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-28 08:53:39,681 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-28 08:53:39,688 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-28 08:53:39,691 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-28 08:53:39,698 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-28 08:53:39,699 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-28 08:53:39,704 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-28 08:53:39,704 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-28 08:53:39,707 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,711 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-28 08:53:39,713 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-28 08:53:39,713 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-28 08:53:39,721 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-28 08:53:39,722 INFO [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-28 08:53:39,724 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.large.0-1682672019723,5,FailOnTimeoutGroup] 2023-04-28 08:53:39,724 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.small.0-1682672019724,5,FailOnTimeoutGroup] 2023-04-28 08:53:39,724 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,724 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-28 08:53:39,726 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,726 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,743 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-28 08:53:39,744 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-28 08:53:39,744 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143 2023-04-28 08:53:39,750 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:34345, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-28 08:53:39,760 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:39,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.ServerManager(394): Registering regionserver=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:39,765 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-28 08:53:39,768 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/info 2023-04-28 08:53:39,769 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-28 08:53:39,771 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:39,771 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-28 08:53:39,774 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/rep_barrier 2023-04-28 08:53:39,774 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-28 08:53:39,775 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:39,775 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-28 08:53:39,778 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/table 2023-04-28 08:53:39,779 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-28 08:53:39,779 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:39,782 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143 2023-04-28 08:53:39,783 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740 2023-04-28 08:53:39,783 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:33889 2023-04-28 08:53:39,783 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-28 08:53:39,784 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740 2023-04-28 08:53:39,788 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:53:39,789 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 08:53:39,789 DEBUG [RS:0;jenkins-hbase6:40909] zookeeper.ZKUtil(162): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:39,790 WARN [RS:0;jenkins-hbase6:40909] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-28 08:53:39,790 INFO [RS:0;jenkins-hbase6:40909] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 08:53:39,791 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1947): logDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/WALs/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:39,792 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-28 08:53:39,792 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase6.apache.org,40909,1682672018517] 2023-04-28 08:53:39,798 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:53:39,799 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64234098, jitterRate=-0.04283735156059265}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:53:39,799 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-28 08:53:39,799 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-28 08:53:39,799 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-28 08:53:39,800 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-28 08:53:39,800 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-28 08:53:39,800 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-28 08:53:39,801 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-28 08:53:39,801 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-28 08:53:39,802 DEBUG [RS:0;jenkins-hbase6:40909] zookeeper.ZKUtil(162): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:39,806 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-28 08:53:39,806 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-28 08:53:39,813 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-28 08:53:39,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-28 08:53:39,822 INFO [RS:0;jenkins-hbase6:40909] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-28 08:53:39,829 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-28 08:53:39,832 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-28 08:53:39,844 INFO [RS:0;jenkins-hbase6:40909] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-28 08:53:39,849 INFO [RS:0;jenkins-hbase6:40909] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-28 08:53:39,849 INFO [RS:0;jenkins-hbase6:40909] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,851 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-28 08:53:39,860 INFO [RS:0;jenkins-hbase6:40909] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,860 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,861 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,861 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,861 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,861 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,862 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase6:0, corePoolSize=2, maxPoolSize=2 2023-04-28 08:53:39,862 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,862 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,862 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,862 DEBUG [RS:0;jenkins-hbase6:40909] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase6:0, corePoolSize=1, maxPoolSize=1 2023-04-28 08:53:39,863 INFO [RS:0;jenkins-hbase6:40909] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,863 INFO [RS:0;jenkins-hbase6:40909] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,864 INFO [RS:0;jenkins-hbase6:40909] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,880 INFO [RS:0;jenkins-hbase6:40909] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-28 08:53:39,882 INFO [RS:0;jenkins-hbase6:40909] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,40909,1682672018517-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:39,900 INFO [RS:0;jenkins-hbase6:40909] regionserver.Replication(203): jenkins-hbase6.apache.org,40909,1682672018517 started 2023-04-28 08:53:39,900 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1638): Serving as jenkins-hbase6.apache.org,40909,1682672018517, RpcServer on jenkins-hbase6.apache.org/172.31.2.20:40909, sessionid=0x100f6e3c0280001 2023-04-28 08:53:39,900 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-28 08:53:39,900 DEBUG [RS:0;jenkins-hbase6:40909] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:39,900 DEBUG [RS:0;jenkins-hbase6:40909] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,40909,1682672018517' 2023-04-28 08:53:39,900 DEBUG [RS:0;jenkins-hbase6:40909] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:39,901 DEBUG [RS:0;jenkins-hbase6:40909] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:39,902 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-28 08:53:39,902 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-28 08:53:39,902 DEBUG [RS:0;jenkins-hbase6:40909] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:39,902 DEBUG [RS:0;jenkins-hbase6:40909] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase6.apache.org,40909,1682672018517' 2023-04-28 08:53:39,902 DEBUG [RS:0;jenkins-hbase6:40909] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-28 08:53:39,902 DEBUG [RS:0;jenkins-hbase6:40909] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-28 08:53:39,903 DEBUG [RS:0;jenkins-hbase6:40909] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-28 08:53:39,903 INFO [RS:0;jenkins-hbase6:40909] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-28 08:53:39,903 INFO [RS:0;jenkins-hbase6:40909] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-28 08:53:39,984 DEBUG [jenkins-hbase6:32801] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-28 08:53:39,987 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase6.apache.org,40909,1682672018517, state=OPENING 2023-04-28 08:53:39,995 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-28 08:53:39,996 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:39,997 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-28 08:53:40,000 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:53:40,019 INFO [RS:0;jenkins-hbase6:40909] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C40909%2C1682672018517, suffix=, logDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/WALs/jenkins-hbase6.apache.org,40909,1682672018517, archiveDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/oldWALs, maxLogs=32 2023-04-28 08:53:40,044 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46297,DS-2cf46bf0-3876-4701-bac8-b95080c59f01,DISK] 2023-04-28 08:53:40,048 INFO [RS:0;jenkins-hbase6:40909] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/WALs/jenkins-hbase6.apache.org,40909,1682672018517/jenkins-hbase6.apache.org%2C40909%2C1682672018517.1682672020023 2023-04-28 08:53:40,051 DEBUG [RS:0;jenkins-hbase6:40909] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46297,DS-2cf46bf0-3876-4701-bac8-b95080c59f01,DISK]] 2023-04-28 08:53:40,191 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:40,194 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-28 08:53:40,201 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58972, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-28 08:53:40,215 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-28 08:53:40,215 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-28 08:53:40,220 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase6.apache.org%2C40909%2C1682672018517.meta, suffix=.meta, logDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/WALs/jenkins-hbase6.apache.org,40909,1682672018517, archiveDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/oldWALs, maxLogs=32 2023-04-28 08:53:40,242 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46297,DS-2cf46bf0-3876-4701-bac8-b95080c59f01,DISK] 2023-04-28 08:53:40,249 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/WALs/jenkins-hbase6.apache.org,40909,1682672018517/jenkins-hbase6.apache.org%2C40909%2C1682672018517.meta.1682672020224.meta 2023-04-28 08:53:40,249 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46297,DS-2cf46bf0-3876-4701-bac8-b95080c59f01,DISK]] 2023-04-28 08:53:40,249 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:53:40,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-28 08:53:40,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-28 08:53:40,276 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-28 08:53:40,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-28 08:53:40,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:40,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-28 08:53:40,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-28 08:53:40,288 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-28 08:53:40,290 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/info 2023-04-28 08:53:40,291 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/info 2023-04-28 08:53:40,291 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-28 08:53:40,293 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:40,293 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-28 08:53:40,295 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/rep_barrier 2023-04-28 08:53:40,295 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/rep_barrier 2023-04-28 08:53:40,295 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-28 08:53:40,297 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:40,297 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-28 08:53:40,303 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/table 2023-04-28 08:53:40,303 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/table 2023-04-28 08:53:40,303 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-28 08:53:40,304 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:40,306 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740 2023-04-28 08:53:40,310 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740 2023-04-28 08:53:40,314 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:53:40,318 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-28 08:53:40,319 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67680880, jitterRate=0.008523702621459961}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:53:40,320 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-28 08:53:40,333 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682672020181 2023-04-28 08:53:40,354 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-28 08:53:40,355 INFO [RS_OPEN_META-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-28 08:53:40,355 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase6.apache.org,40909,1682672018517, state=OPEN 2023-04-28 08:53:40,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-28 08:53:40,358 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-28 08:53:40,363 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-28 08:53:40,363 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase6.apache.org,40909,1682672018517 in 358 msec 2023-04-28 08:53:40,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-28 08:53:40,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 548 msec 2023-04-28 08:53:40,374 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 801 msec 2023-04-28 08:53:40,375 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682672020375, completionTime=-1 2023-04-28 08:53:40,375 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-28 08:53:40,376 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-28 08:53:40,460 DEBUG [hconnection-0x52f5cfe6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:40,464 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58976, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:40,481 INFO [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-28 08:53:40,482 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682672080482 2023-04-28 08:53:40,482 INFO [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682672140482 2023-04-28 08:53:40,482 INFO [master/jenkins-hbase6:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 106 msec 2023-04-28 08:53:40,505 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,32801,1682672017275-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:40,506 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,32801,1682672017275-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:40,506 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,32801,1682672017275-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:40,508 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase6:32801, period=300000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:40,508 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-28 08:53:40,514 DEBUG [master/jenkins-hbase6:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-28 08:53:40,523 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-28 08:53:40,525 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-28 08:53:40,537 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-28 08:53:40,540 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 08:53:40,544 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 08:53:40,569 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:40,571 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb empty. 2023-04-28 08:53:40,572 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:40,572 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-28 08:53:41,012 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-28 08:53:41,015 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => a999af13cd42d26d9cb3ed05ba7639cb, NAME => 'hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp 2023-04-28 08:53:41,029 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:41,029 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing a999af13cd42d26d9cb3ed05ba7639cb, disabling compactions & flushes 2023-04-28 08:53:41,030 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:53:41,030 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:53:41,030 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. after waiting 0 ms 2023-04-28 08:53:41,030 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:53:41,030 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:53:41,030 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for a999af13cd42d26d9cb3ed05ba7639cb: 2023-04-28 08:53:41,035 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 08:53:41,051 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682672021038"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672021038"}]},"ts":"1682672021038"} 2023-04-28 08:53:41,084 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 08:53:41,086 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 08:53:41,091 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672021086"}]},"ts":"1682672021086"} 2023-04-28 08:53:41,096 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-28 08:53:41,106 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a999af13cd42d26d9cb3ed05ba7639cb, ASSIGN}] 2023-04-28 08:53:41,109 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=a999af13cd42d26d9cb3ed05ba7639cb, ASSIGN 2023-04-28 08:53:41,111 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=a999af13cd42d26d9cb3ed05ba7639cb, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=false 2023-04-28 08:53:41,263 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a999af13cd42d26d9cb3ed05ba7639cb, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:41,263 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682672021262"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672021262"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672021262"}]},"ts":"1682672021262"} 2023-04-28 08:53:41,268 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure a999af13cd42d26d9cb3ed05ba7639cb, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:53:41,428 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:53:41,429 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a999af13cd42d26d9cb3ed05ba7639cb, NAME => 'hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:53:41,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:41,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:41,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:41,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:41,432 INFO [StoreOpener-a999af13cd42d26d9cb3ed05ba7639cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:41,434 DEBUG [StoreOpener-a999af13cd42d26d9cb3ed05ba7639cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/info 2023-04-28 08:53:41,434 DEBUG [StoreOpener-a999af13cd42d26d9cb3ed05ba7639cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/info 2023-04-28 08:53:41,435 INFO [StoreOpener-a999af13cd42d26d9cb3ed05ba7639cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a999af13cd42d26d9cb3ed05ba7639cb columnFamilyName info 2023-04-28 08:53:41,436 INFO [StoreOpener-a999af13cd42d26d9cb3ed05ba7639cb-1] regionserver.HStore(310): Store=a999af13cd42d26d9cb3ed05ba7639cb/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:41,437 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:41,438 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:41,444 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:53:41,447 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:53:41,449 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened a999af13cd42d26d9cb3ed05ba7639cb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71512139, jitterRate=0.06561391055583954}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-28 08:53:41,449 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for a999af13cd42d26d9cb3ed05ba7639cb: 2023-04-28 08:53:41,451 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb., pid=6, masterSystemTime=1682672021422 2023-04-28 08:53:41,455 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:53:41,455 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:53:41,456 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=a999af13cd42d26d9cb3ed05ba7639cb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:41,457 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682672021456"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672021456"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672021456"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672021456"}]},"ts":"1682672021456"} 2023-04-28 08:53:41,464 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-28 08:53:41,464 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure a999af13cd42d26d9cb3ed05ba7639cb, server=jenkins-hbase6.apache.org,40909,1682672018517 in 193 msec 2023-04-28 08:53:41,468 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-28 08:53:41,468 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=a999af13cd42d26d9cb3ed05ba7639cb, ASSIGN in 358 msec 2023-04-28 08:53:41,469 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 08:53:41,470 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672021470"}]},"ts":"1682672021470"} 2023-04-28 08:53:41,473 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-28 08:53:41,477 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 08:53:41,480 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 953 msec 2023-04-28 08:53:41,540 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-28 08:53:41,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-28 08:53:41,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:53:41,584 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-28 08:53:41,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-28 08:53:41,607 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-04-28 08:53:41,619 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-28 08:53:41,633 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-28 08:53:41,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-04-28 08:53:41,645 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-28 08:53:41,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-28 08:53:41,650 INFO [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.041sec 2023-04-28 08:53:41,653 INFO [master/jenkins-hbase6:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-28 08:53:41,655 INFO [master/jenkins-hbase6:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-28 08:53:41,655 INFO [master/jenkins-hbase6:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-28 08:53:41,657 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,32801,1682672017275-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-28 08:53:41,657 INFO [master/jenkins-hbase6:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase6.apache.org,32801,1682672017275-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-28 08:53:41,669 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-28 08:53:41,671 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:41,686 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:41,706 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:41,718 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58982, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:41,730 INFO [Listener at localhost/39717] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:53:41,752 INFO [Listener at localhost/39717] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=336, ProcessCount=172, AvailableMemoryMB=7016 2023-04-28 08:53:41,775 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 08:53:41,779 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:54516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 08:53:41,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:53:41,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:53:41,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 08:53:41,797 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 08:53:41,799 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 08:53:41,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-28 08:53:41,802 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:41,804 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796 empty. 2023-04-28 08:53:41,805 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:41,805 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:53:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 08:53:41,822 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:53:41,825 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6dfeafaee81d2f031bddf09740078796, NAME => 'TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp 2023-04-28 08:53:41,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:41,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6dfeafaee81d2f031bddf09740078796, disabling compactions & flushes 2023-04-28 08:53:41,838 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:41,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:41,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. after waiting 0 ms 2023-04-28 08:53:41,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:41,838 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:41,838 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:41,843 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 08:53:41,845 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672021845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672021845"}]},"ts":"1682672021845"} 2023-04-28 08:53:41,848 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 08:53:41,850 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 08:53:41,850 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672021850"}]},"ts":"1682672021850"} 2023-04-28 08:53:41,853 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 08:53:41,858 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dfeafaee81d2f031bddf09740078796, ASSIGN}] 2023-04-28 08:53:41,860 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dfeafaee81d2f031bddf09740078796, ASSIGN 2023-04-28 08:53:41,862 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dfeafaee81d2f031bddf09740078796, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=false 2023-04-28 08:53:41,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 08:53:42,013 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6dfeafaee81d2f031bddf09740078796, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:42,014 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672022013"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672022013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672022013"}]},"ts":"1682672022013"} 2023-04-28 08:53:42,018 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:53:42,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 08:53:42,178 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:42,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6dfeafaee81d2f031bddf09740078796, NAME => 'TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:53:42,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:53:42,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,181 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,184 DEBUG [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A 2023-04-28 08:53:42,184 DEBUG [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A 2023-04-28 08:53:42,186 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:53:42,186 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6dfeafaee81d2f031bddf09740078796 columnFamilyName A 2023-04-28 08:53:42,187 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.HStore(310): Store=6dfeafaee81d2f031bddf09740078796/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:42,187 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,189 DEBUG [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B 2023-04-28 08:53:42,189 DEBUG [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B 2023-04-28 08:53:42,190 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:53:42,190 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6dfeafaee81d2f031bddf09740078796 columnFamilyName B 2023-04-28 08:53:42,191 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.HStore(310): Store=6dfeafaee81d2f031bddf09740078796/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:42,191 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,193 DEBUG [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C 2023-04-28 08:53:42,193 DEBUG [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C 2023-04-28 08:53:42,194 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:53:42,194 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6dfeafaee81d2f031bddf09740078796 columnFamilyName C 2023-04-28 08:53:42,195 INFO [StoreOpener-6dfeafaee81d2f031bddf09740078796-1] regionserver.HStore(310): Store=6dfeafaee81d2f031bddf09740078796/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:53:42,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:42,197 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:53:42,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:53:42,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 6dfeafaee81d2f031bddf09740078796; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68590777, jitterRate=0.022082224488258362}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:53:42,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:42,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., pid=11, masterSystemTime=1682672022171 2023-04-28 08:53:42,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:42,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:42,214 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6dfeafaee81d2f031bddf09740078796, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:42,214 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672022214"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672022214"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672022214"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672022214"}]},"ts":"1682672022214"} 2023-04-28 08:53:42,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-28 08:53:42,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 in 199 msec 2023-04-28 08:53:42,223 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-28 08:53:42,224 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dfeafaee81d2f031bddf09740078796, ASSIGN in 362 msec 2023-04-28 08:53:42,225 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 08:53:42,225 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672022225"}]},"ts":"1682672022225"} 2023-04-28 08:53:42,227 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 08:53:42,230 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 08:53:42,232 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 437 msec 2023-04-28 08:53:42,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-28 08:53:42,424 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-28 08:53:42,429 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,434 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,436 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,440 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58998, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,443 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 08:53:42,446 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:54528, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 08:53:42,456 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,461 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,462 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,469 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,469 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,478 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,478 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,483 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,483 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,488 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,488 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,493 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,494 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,531 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,532 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,536 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,537 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:53:42,542 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:53:42,556 DEBUG [hconnection-0x74949e48-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,562 DEBUG [hconnection-0x1a5dd2ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,567 DEBUG [hconnection-0x3781157c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:42,567 DEBUG [hconnection-0x5de3642b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,567 DEBUG [hconnection-0x2b0f04f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,567 DEBUG [hconnection-0x6b828d38-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,570 DEBUG [hconnection-0x7e007515-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,570 DEBUG [hconnection-0x3685aa1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,570 DEBUG [hconnection-0x742302d5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:53:42,583 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,586 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,592 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,592 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,600 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,600 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59032, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,601 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59062, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,602 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,602 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:53:42,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:42,612 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:42,612 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:42,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:42,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:42,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:42,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:42,625 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:42,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:42,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:42,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:42,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:42,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:42,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:42,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:42,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:42,630 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:42,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:42,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:42,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:42,633 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:42,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:42,635 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:42,643 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 08:53:42,657 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:42,658 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:42,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:42,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:42,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:42,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:42,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:42,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:42,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672022747, max timestamp=1682672022781Num uniques 859; 2023-04-28 08:53:42,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:53:42,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682672022747, max timestamp=1682672022781Num uniques 859; 2023-04-28 08:53:42,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 08:53:42,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672022747, max timestamp=1682672022793Num uniques 859; 2023-04-28 08:53:42,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:53:42,876 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/b5334359a7da4758a0fae409644e23e0 2023-04-28 08:53:43,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672082981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682672022966, max timestamp=1682672022976Num uniques 1018; 2023-04-28 08:53:43,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-28 08:53:43,006 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682672022966, max timestamp=1682672022978Num uniques 1018; 2023-04-28 08:53:43,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 08:53:43,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682672022966, max timestamp=1682672022976Num uniques 1009; 2023-04-28 08:53:43,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-28 08:53:43,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672023035, max timestamp=1682672023053Num uniques 1009; 2023-04-28 08:53:43,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10392, suffix heap size=132904,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=227 2023-04-28 08:53:43,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682672023037, max timestamp=1682672023054Num uniques 1027; 2023-04-28 08:53:43,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-28 08:53:43,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682672023037, max timestamp=1682672023053Num uniques 1009; 2023-04-28 08:53:43,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-28 08:53:43,104 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/0c96ddf530424995b6ccb78b33e6fd44 2023-04-28 08:53:43,170 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/9f64ded55ee64abfb740ed9246556efb 2023-04-28 08:53:43,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1682672023146, max timestamp=1682672023168Num uniques 1009; 2023-04-28 08:53:43,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1682672023146, max timestamp=1682672023167Num uniques 1009; 2023-04-28 08:53:43,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=8506, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-28 08:53:43,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-28 08:53:43,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1682672023146, max timestamp=1682672023168Num uniques 1086; 2023-04-28 08:53:43,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=49726, new segment data size=8506, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-28 08:53:43,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/b5334359a7da4758a0fae409644e23e0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b5334359a7da4758a0fae409644e23e0 2023-04-28 08:53:43,202 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b5334359a7da4758a0fae409644e23e0, entries=100, sequenceid=7, filesize=9.3 K 2023-04-28 08:53:43,218 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/0c96ddf530424995b6ccb78b33e6fd44 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0c96ddf530424995b6ccb78b33e6fd44 2023-04-28 08:53:43,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682672023201, max timestamp=1682672023211Num uniques 1045; 2023-04-28 08:53:43,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682672023201, max timestamp=1682672023211Num uniques 1045; 2023-04-28 08:53:43,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1682672023201, max timestamp=1682672023211Num uniques 1045; 2023-04-28 08:53:43,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-28 08:53:43,224 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-28 08:53:43,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-28 08:53:43,254 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0c96ddf530424995b6ccb78b33e6fd44, entries=100, sequenceid=7, filesize=9.3 K 2023-04-28 08:53:43,265 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/9f64ded55ee64abfb740ed9246556efb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9f64ded55ee64abfb740ed9246556efb 2023-04-28 08:53:43,279 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9f64ded55ee64abfb740ed9246556efb, entries=100, sequenceid=7, filesize=9.3 K 2023-04-28 08:53:43,285 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=53.67 KB/54960 for 6dfeafaee81d2f031bddf09740078796 in 649ms, sequenceid=7, compaction requested=false 2023-04-28 08:53:43,293 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 08:53:43,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:43,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:43,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:43,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:43,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:43,304 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:43,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:43,307 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=60.38 KB heapSize=157.78 KB 2023-04-28 08:53:43,307 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:43,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:43,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:43,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:43,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:43,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:43,309 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,310 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,310 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:43,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:43,330 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:43,330 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@277113ac[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:43,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:43,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:43,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:43,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:43,357 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:43,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:43,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:43,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:43,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:43,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:43,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:43,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:43,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:43,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,400 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:43,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:43,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:43,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:43,439 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:43,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:43,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:43,447 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:43,447 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:43,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:43,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672083463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672083467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672083481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672083499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672023455, max timestamp=1682672023499Num uniques 859; 2023-04-28 08:53:43,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:53:43,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672023455, max timestamp=1682672023499Num uniques 859; 2023-04-28 08:53:43,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672023455, max timestamp=1682672023499Num uniques 859; 2023-04-28 08:53:43,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:53:43,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:53:43,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:43,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:43,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:43,587 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:43,587 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:43,589 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:43,590 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:43,590 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,590 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,592 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:43,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:43,592 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,593 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,593 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:43,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:43,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:43,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:43,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:43,600 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:43,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:43,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:43,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:43,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:43,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:43,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:43,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:43,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:43,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:43,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:43,615 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,618 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:43,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39afc4a6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:43,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:43,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,622 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,623 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:43,623 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:43,623 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:43,623 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,623 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:43,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:43,626 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:43,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:43,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:43,636 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:43,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:43,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:43,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:43,636 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:43,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:43,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,645 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,649 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672083644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672083644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672083648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672083650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672083651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:43,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:43,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:43,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:43,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:43,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:43,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,674 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:43,675 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:43,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:43,775 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:43,775 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672083770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672083770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:43,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672083771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672083772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672083771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:43,812 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:43,812 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:43,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:43,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:43,815 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,815 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:43,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:43,817 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:43,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:43,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:43,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:43,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:43,821 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:43,821 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:43,822 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:43,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:43,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:43,828 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:43,829 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:43,829 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:43,829 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:43,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:43,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:43,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,832 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:43,832 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:43,837 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:43,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f288152[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:43,842 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:43,842 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,844 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:43,845 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,845 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:43,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:43,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:43,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:43,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:43,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:43,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/aa7c555d7fe547aa9e7475240280e376 2023-04-28 08:53:43,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:43,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:43,882 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:43,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:43,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:43,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:43,882 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:43,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:43,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,888 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:43,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:43,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:43,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:43,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:43,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:43,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:43,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:43,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,915 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:43,916 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,919 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:43,919 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:43,919 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:43,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:43,919 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:43,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:43,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:43,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672083979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672083981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672083981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:43,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672083984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:43,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672083984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,019 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:44,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:44,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:44,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:44,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:44,039 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:44,039 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,042 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:44,042 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:44,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,045 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,045 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,046 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,046 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:44,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:44,048 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:44,048 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:44,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:44,052 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:44,052 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,052 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:44,052 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:44,052 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:44,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:44,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:44,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:44,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,058 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,058 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb3e288[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:44,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:44,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,062 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:44,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:44,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:44,063 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,063 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:44,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:44,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:44,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:44,076 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:44,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:44,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:44,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:44,076 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,084 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,109 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,109 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,110 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:44,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:44,110 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,111 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,112 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:44,112 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:44,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:44,212 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:44,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:44,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:44,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:44,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:44,239 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:44,239 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:44,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:44,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,247 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,247 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,247 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:44,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:44,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:44,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:44,249 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:44,249 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,251 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:44,253 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:44,253 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:44,253 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,253 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:44,253 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:44,253 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:44,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:44,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:44,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:44,258 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f074a1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:44,261 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:44,261 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,263 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,264 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:44,264 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:44,264 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:44,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:44,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:44,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:44,275 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:44,275 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:44,276 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:44,276 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:44,276 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:44,278 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,278 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,278 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,284 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672084287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672084288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672084288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672084291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672084293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:44,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:44,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:44,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,301 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:44,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:44,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/187bb1f5e059489599aaef3eb180e649 2023-04-28 08:53:44,402 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:44,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:44,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:44,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:44,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:44,423 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:44,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:44,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:44,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,428 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,430 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,430 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,430 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:44,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:44,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:44,437 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:44,437 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:44,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:44,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:44,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:44,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:44,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:44,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:44,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:44,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:44,444 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273d16ed[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:44,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:44,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:44,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:44,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:44,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:44,455 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:44,455 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,455 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:44,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:44,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:44,463 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:44,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:44,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:44,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:44,463 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:44,465 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,465 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,466 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,472 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,476 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,476 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,476 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,476 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,476 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:44,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:44,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,477 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:44,478 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:44,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:44,578 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:44,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:44,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:44,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:44,607 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:44,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:44,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:44,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,612 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,614 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,614 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:44,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:44,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:44,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:44,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:44,630 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:44,630 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:44,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:44,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:44,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:44,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:44,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:44,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:44,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:44,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:44,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c743149[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:44,645 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:44,645 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:44,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:44,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:44,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:44,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,651 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:44,656 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:44,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:44,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:44,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:44,657 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:44,668 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,668 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,668 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:44,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:44,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,681 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:44,682 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:44,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,782 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:44,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:44,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:44,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672084791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672084793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672084793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:44,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:44,800 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:44,800 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672084798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:44,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:44,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,805 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:44,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672084805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,806 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:44,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:44,809 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:44,809 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:44,809 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:44,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:44,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:44,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:44,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:44,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:44,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:44,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:44,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,817 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:44,817 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:44,819 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,819 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,820 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,820 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:44,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c032ded[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:44,824 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:44,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/9eaa88499e0a41128b112ec7675f277d 2023-04-28 08:53:44,824 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,828 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:44,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:44,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:44,829 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,829 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:44,832 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:44,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/aa7c555d7fe547aa9e7475240280e376 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa7c555d7fe547aa9e7475240280e376 2023-04-28 08:53:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:44,844 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:44,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:44,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:44,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:44,844 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:44,846 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,846 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,846 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:44,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:44,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa7c555d7fe547aa9e7475240280e376, entries=200, sequenceid=100, filesize=13.9 K 2023-04-28 08:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/187bb1f5e059489599aaef3eb180e649 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/187bb1f5e059489599aaef3eb180e649 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:44,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:44,870 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:44,870 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:44,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:44,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:44,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:44,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/187bb1f5e059489599aaef3eb180e649, entries=150, sequenceid=100, filesize=11.6 K 2023-04-28 08:53:44,890 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/9eaa88499e0a41128b112ec7675f277d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9eaa88499e0a41128b112ec7675f277d 2023-04-28 08:53:44,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9eaa88499e0a41128b112ec7675f277d, entries=150, sequenceid=100, filesize=11.6 K 2023-04-28 08:53:44,926 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~157.73 KB/161520, currentSize=127.47 KB/130530 for 6dfeafaee81d2f031bddf09740078796 in 1619ms, sequenceid=100, compaction requested=false 2023-04-28 08:53:44,931 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:44,970 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:44,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:44,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:44,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:44,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:44,991 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:44,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:44,992 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:44,992 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:44,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,995 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:44,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:44,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:44,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:44,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:44,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:44,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:44,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:44,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:45,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:45,013 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:45,013 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:45,013 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:45,013 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:45,013 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-28 08:53:45,014 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:45,015 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:45,015 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:45,015 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:45,015 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:45,015 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:45,078 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/a4caf73cba314f518eff75bee39d81c5 2023-04-28 08:53:45,547 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/03ba4d03b92c4f13a38d98946fd1c523 2023-04-28 08:53:45,605 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/28752df514fd41cb80173dde7a23cb25 2023-04-28 08:53:45,622 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/a4caf73cba314f518eff75bee39d81c5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/a4caf73cba314f518eff75bee39d81c5 2023-04-28 08:53:45,640 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/a4caf73cba314f518eff75bee39d81c5, entries=150, sequenceid=134, filesize=11.8 K 2023-04-28 08:53:45,648 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/03ba4d03b92c4f13a38d98946fd1c523 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/03ba4d03b92c4f13a38d98946fd1c523 2023-04-28 08:53:45,675 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/03ba4d03b92c4f13a38d98946fd1c523, entries=150, sequenceid=134, filesize=11.8 K 2023-04-28 08:53:45,680 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/28752df514fd41cb80173dde7a23cb25 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28752df514fd41cb80173dde7a23cb25 2023-04-28 08:53:45,701 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28752df514fd41cb80173dde7a23cb25, entries=150, sequenceid=134, filesize=11.8 K 2023-04-28 08:53:45,704 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 6dfeafaee81d2f031bddf09740078796 in 691ms, sequenceid=134, compaction requested=true 2023-04-28 08:53:45,704 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:45,704 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:45,704 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:45,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:45,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:45,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:45,710 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:45,710 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:45,711 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,711 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3507b0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:45,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:45,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,717 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,717 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:45,717 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:45,717 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:45,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:45,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:45,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:45,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:45,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:45,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:45,726 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:45,726 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:45,726 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:45,726 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:45,726 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:45,731 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,731 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:45,731 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:45,732 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,738 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:45,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:45,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:45,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,742 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:45,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:45,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:45,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:45,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,743 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:45,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:45,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:45,744 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:45,744 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:45,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:45,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:53:45,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:45,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:45,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:45,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:45,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:45,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:45,844 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:45,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:45,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:45,869 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 08:53:45,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:45,885 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:45,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:45,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:45,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:45,892 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:45,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672025852, max timestamp=1682672025885Num uniques 859; 2023-04-28 08:53:45,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:53:45,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672025852, max timestamp=1682672025885Num uniques 859; 2023-04-28 08:53:45,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672025852, max timestamp=1682672025885Num uniques 859; 2023-04-28 08:53:45,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:53:45,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:53:45,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:45,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:45,902 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,902 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:45,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:45,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:45,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:45,908 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:45,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:45,926 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:45,926 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:45,926 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:45,926 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:45,926 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:45,926 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:45,926 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:45,926 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:45,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:45,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:45,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:45,930 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:45,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30f0fcc5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:45,933 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:45,933 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,935 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:45,935 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:45,935 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:45,936 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,936 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:45,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:45,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:45,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:45,941 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:45,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:45,946 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:45,946 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:45,946 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:45,946 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:45,946 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:45,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:45,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,953 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,962 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:45,962 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:45,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:45,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:45,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:45,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:45,962 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:45,963 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:45,963 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:45,963 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:45,963 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,963 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:45,963 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:45,963 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:45,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:45,963 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:45,989 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-28 08:53:45,992 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-28 08:53:46,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672086013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672086014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672086024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672086027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672086024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,063 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:46,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:46,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:46,074 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:46,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:46,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:46,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,078 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,078 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:46,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:46,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:46,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:46,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,085 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:46,086 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:46,086 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:46,086 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,086 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:46,086 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:46,086 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,086 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:46,086 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:46,086 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,086 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:46,086 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:46,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,089 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,091 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,091 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:46,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d637deb[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:46,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:46,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,096 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,096 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:46,096 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:46,096 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:46,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:46,104 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:46,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,104 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,104 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:46,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:46,113 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:46,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:46,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:46,114 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:46,114 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:46,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,121 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,125 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:46,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:46,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,125 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,127 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:46,127 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:46,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:46,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672086130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672086131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672086138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672086138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672086140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,227 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:46,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:46,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:46,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:46,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:46,240 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:46,240 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:46,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:46,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,250 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,251 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:46,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:46,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:46,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:46,253 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:46,253 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,255 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:46,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:46,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:46,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:46,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:46,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:46,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:46,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:46,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:46,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:46,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@580a5fef[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:46,269 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:46,269 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,270 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,271 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:46,271 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:46,271 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:46,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:46,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:46,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:46,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:46,292 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:46,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:46,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:46,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:46,292 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:46,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,300 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/6cfdb64ebe404325a2ac9f0cec577ce2 2023-04-28 08:53:46,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,302 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,310 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,310 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,326 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:46,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:46,328 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:46,328 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:46,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672086336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672086339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672086342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672086344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672086355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,428 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:46,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:46,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:46,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:46,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:46,439 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:46,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:46,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:46,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,447 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:46,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:46,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:46,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:46,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:46,464 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:46,464 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:46,468 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,468 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:46,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:46,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:46,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:46,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:46,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:46,472 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,473 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,473 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,473 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:46,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:46,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd5dbd9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:46,476 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:46,476 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,478 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:46,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:46,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:46,479 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,479 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:46,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:46,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:46,483 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:46,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:46,487 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:46,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:46,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:46,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:46,487 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:46,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:46,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:46,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,509 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,509 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,509 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,509 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:46,509 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:46,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,609 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:46,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:46,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:46,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:46,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:46,619 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:46,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:46,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:46,621 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,623 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,623 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,624 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,624 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:46,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:46,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:46,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:46,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:46,627 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:46,627 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:46,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:46,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:46,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:46,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:46,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:46,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:46,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:46,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:46,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,635 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:46,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38740832[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:46,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:46,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:46,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:46,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:46,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:46,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:46,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672086645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672086646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672086647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,659 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:46,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:46,661 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:46,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:46,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:46,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:46,661 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:46,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672086658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,663 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,664 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,664 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:46,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672086663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,675 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:46,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:46,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:46,678 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:46,678 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:46,778 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:46,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:46,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:46,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:46,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:46,786 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:46,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:46,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:46,789 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,789 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,791 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:46,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:46,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:46,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:46,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:46,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:46,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:46,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:46,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:46,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:46,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:46,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:46,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:46,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:46,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ebea6f2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:46,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:46,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:46,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/e6e4372cfb8d439eb0c2b82d439611c2 2023-04-28 08:53:46,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:46,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:46,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:46,834 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,835 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:46,835 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,835 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:46,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:46,846 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:46,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:46,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:46,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:46,846 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:46,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:46,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:46,857 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:46,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:46,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:46,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:46,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:46,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:46,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,868 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,868 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:46,868 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,868 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:46,868 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:46,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:46,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:46,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:46,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/d9c70a9cef9041caabd24c6c5764e111 2023-04-28 08:53:46,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/6cfdb64ebe404325a2ac9f0cec577ce2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/6cfdb64ebe404325a2ac9f0cec577ce2 2023-04-28 08:53:46,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/6cfdb64ebe404325a2ac9f0cec577ce2, entries=200, sequenceid=144, filesize=14.1 K 2023-04-28 08:53:46,908 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/e6e4372cfb8d439eb0c2b82d439611c2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e6e4372cfb8d439eb0c2b82d439611c2 2023-04-28 08:53:46,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e6e4372cfb8d439eb0c2b82d439611c2, entries=150, sequenceid=144, filesize=11.8 K 2023-04-28 08:53:46,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/d9c70a9cef9041caabd24c6c5764e111 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d9c70a9cef9041caabd24c6c5764e111 2023-04-28 08:53:46,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d9c70a9cef9041caabd24c6c5764e111, entries=150, sequenceid=144, filesize=11.8 K 2023-04-28 08:53:46,938 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 6dfeafaee81d2f031bddf09740078796 in 1114ms, sequenceid=144, compaction requested=true 2023-04-28 08:53:46,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:46,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:46,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:46,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:46,941 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:53:46,941 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:53:46,951 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50230 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:53:46,951 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45500 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:53:46,953 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:46,953 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:46,953 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,953 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:46,953 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0c96ddf530424995b6ccb78b33e6fd44, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/187bb1f5e059489599aaef3eb180e649, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/03ba4d03b92c4f13a38d98946fd1c523, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e6e4372cfb8d439eb0c2b82d439611c2] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=44.4 K 2023-04-28 08:53:46,953 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b5334359a7da4758a0fae409644e23e0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa7c555d7fe547aa9e7475240280e376, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/a4caf73cba314f518eff75bee39d81c5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/6cfdb64ebe404325a2ac9f0cec577ce2] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=49.1 K 2023-04-28 08:53:46,955 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b5334359a7da4758a0fae409644e23e0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682672022607 2023-04-28 08:53:46,955 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0c96ddf530424995b6ccb78b33e6fd44, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682672022607 2023-04-28 08:53:46,958 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting aa7c555d7fe547aa9e7475240280e376, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1682672023205 2023-04-28 08:53:46,958 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 187bb1f5e059489599aaef3eb180e649, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1682672023205 2023-04-28 08:53:46,960 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 03ba4d03b92c4f13a38d98946fd1c523, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1682672023639 2023-04-28 08:53:46,960 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a4caf73cba314f518eff75bee39d81c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1682672023639 2023-04-28 08:53:46,961 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e6e4372cfb8d439eb0c2b82d439611c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682672025816 2023-04-28 08:53:46,962 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 6cfdb64ebe404325a2ac9f0cec577ce2, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682672025808 2023-04-28 08:53:46,968 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:46,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:46,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:46,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:46,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:46,978 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:46,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:46,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:46,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:46,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,983 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:46,983 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,984 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:46,984 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:46,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:46,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:46,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:46,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:46,998 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:46,998 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:47,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:47,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:47,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:47,003 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:47,003 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:47,003 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-28 08:53:47,003 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:47,004 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:47,004 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:47,004 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:47,004 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:47,004 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:47,005 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#12 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:47,025 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#13 average throughput is 0.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:47,112 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/5245b57038024a18bfe60af8450b59df as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/5245b57038024a18bfe60af8450b59df 2023-04-28 08:53:47,137 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/17756b75d65840a794a7a45451fabf4a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/17756b75d65840a794a7a45451fabf4a 2023-04-28 08:53:47,167 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into 5245b57038024a18bfe60af8450b59df(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:47,167 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:47,168 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=12, startTime=1682672026941; duration=0sec 2023-04-28 08:53:47,168 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:47,169 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:53:47,171 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45500 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:53:47,171 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:53:47,171 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:47,172 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9f64ded55ee64abfb740ed9246556efb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9eaa88499e0a41128b112ec7675f277d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28752df514fd41cb80173dde7a23cb25, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d9c70a9cef9041caabd24c6c5764e111] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=44.4 K 2023-04-28 08:53:47,172 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9f64ded55ee64abfb740ed9246556efb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682672022607 2023-04-28 08:53:47,173 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9eaa88499e0a41128b112ec7675f277d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1682672023205 2023-04-28 08:53:47,173 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 28752df514fd41cb80173dde7a23cb25, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1682672023639 2023-04-28 08:53:47,174 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d9c70a9cef9041caabd24c6c5764e111, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682672025816 2023-04-28 08:53:47,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:47,180 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:47,185 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into 17756b75d65840a794a7a45451fabf4a(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:47,186 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:47,186 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=12, startTime=1682672026941; duration=0sec 2023-04-28 08:53:47,186 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:47,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,210 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#15 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:47,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672087203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672087201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672087210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672087211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672087211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,316 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672087315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672087320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672087327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672087331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672087344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,448 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-28 08:53:47,449 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-28 08:53:47,505 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/4ee24e314e2e438390ff91df14a92bf2 2023-04-28 08:53:47,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672087523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672087524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672087533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672087539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672087551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,557 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/75730495d9544463a37fd901e88d204d 2023-04-28 08:53:47,588 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/2d5faae90a6343d9ba8bfb92876399be 2023-04-28 08:53:47,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/4ee24e314e2e438390ff91df14a92bf2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4ee24e314e2e438390ff91df14a92bf2 2023-04-28 08:53:47,611 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4ee24e314e2e438390ff91df14a92bf2, entries=150, sequenceid=181, filesize=11.8 K 2023-04-28 08:53:47,614 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/75730495d9544463a37fd901e88d204d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/75730495d9544463a37fd901e88d204d 2023-04-28 08:53:47,627 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/75730495d9544463a37fd901e88d204d, entries=150, sequenceid=181, filesize=11.8 K 2023-04-28 08:53:47,630 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/2d5faae90a6343d9ba8bfb92876399be as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/2d5faae90a6343d9ba8bfb92876399be 2023-04-28 08:53:47,643 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/2d5faae90a6343d9ba8bfb92876399be, entries=150, sequenceid=181, filesize=11.8 K 2023-04-28 08:53:47,645 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=60.38 KB/61830 for 6dfeafaee81d2f031bddf09740078796 in 642ms, sequenceid=181, compaction requested=false 2023-04-28 08:53:47,646 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:47,646 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:47,646 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:47,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:47,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:47,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:47,649 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:47,649 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,649 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74417bed[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:47,652 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:47,652 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,653 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,654 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:47,654 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:47,654 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:47,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:47,655 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:47,655 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:47,655 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:47,656 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:47,664 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:47,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:47,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:47,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:47,664 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:47,666 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,666 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,667 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:47,667 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:47,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:47,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:47,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:47,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:47,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:47,678 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/8b35c53f1c2e43fb9d96e3673b1a1a51 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8b35c53f1c2e43fb9d96e3673b1a1a51 2023-04-28 08:53:47,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:47,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:47,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:47,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,679 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:47,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:47,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:47,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:47,684 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:47,684 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:47,695 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into 8b35c53f1c2e43fb9d96e3673b1a1a51(size=11.9 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:47,695 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:47,695 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=12, startTime=1682672026941; duration=0sec 2023-04-28 08:53:47,695 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:47,784 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:47,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:47,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:47,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:47,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:47,793 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:47,793 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:47,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:47,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:47,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:47,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:47,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:47,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:47,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:47,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:47,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:47,803 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:47,803 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:47,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:47,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:47,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:47,804 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:47,805 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:47,805 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:53:47,805 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:47,806 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:47,806 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:47,806 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:47,806 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:47,806 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:47,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:47,831 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:47,868 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/f88e2c5e0f014fb2b1d6c8b75c76e1de 2023-04-28 08:53:47,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4442) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4381) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4857) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4851) at org.apache.hadoop.hbase.regionserver.HRegion.doBatchMutate(HRegion.java:4847) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3159) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672087881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672087882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672087884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672087886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:47,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672087907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672088013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672088016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672088017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672088017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672088017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672088218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672088225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672088225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672088225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672088226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,309 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/0fed88a0f2074303913b959a0d301671 2023-04-28 08:53:48,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672088522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672088529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672088530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672088530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:48,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672088531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,773 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/1a3b4362c4324c1db12393773f8d3bb9 2023-04-28 08:53:48,782 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/f88e2c5e0f014fb2b1d6c8b75c76e1de as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f88e2c5e0f014fb2b1d6c8b75c76e1de 2023-04-28 08:53:48,801 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f88e2c5e0f014fb2b1d6c8b75c76e1de, entries=150, sequenceid=196, filesize=11.8 K 2023-04-28 08:53:48,804 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/0fed88a0f2074303913b959a0d301671 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0fed88a0f2074303913b959a0d301671 2023-04-28 08:53:48,819 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0fed88a0f2074303913b959a0d301671, entries=150, sequenceid=196, filesize=11.8 K 2023-04-28 08:53:48,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/1a3b4362c4324c1db12393773f8d3bb9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1a3b4362c4324c1db12393773f8d3bb9 2023-04-28 08:53:48,836 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1a3b4362c4324c1db12393773f8d3bb9, entries=150, sequenceid=196, filesize=11.8 K 2023-04-28 08:53:48,837 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6dfeafaee81d2f031bddf09740078796 in 1032ms, sequenceid=196, compaction requested=true 2023-04-28 08:53:48,838 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:48,839 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:48,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:48,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:48,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:48,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:48,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,842 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:48,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:48,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:48,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:48,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@556b6daf[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:48,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:48,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,849 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:48,849 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:48,849 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:48,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:48,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:48,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:48,856 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:48,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:48,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:48,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:48,856 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:48,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:48,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:48,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:48,864 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:48,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:48,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:48,868 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:48,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:48,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:48,869 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:48,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:48,869 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:48,969 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:48,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:48,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:48,975 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:48,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:48,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:48,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:48,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:48,982 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:48,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:48,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:48,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:48,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:48,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:48,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:48,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:48,984 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:48,984 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:48,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:48,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:48,986 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:48,986 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:48,986 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:48,986 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:53:48,986 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:48,986 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:48,986 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:48,987 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:48,987 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:48,987 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:49,019 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/81422f1c28ff48088bb88343cabf36db 2023-04-28 08:53:49,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:49,030 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:49,055 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/809bd19ab67c4d4b8b626f4bc84e0eab 2023-04-28 08:53:49,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672089052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672089055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672089060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672089062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672089065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,093 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=216 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/8efed1ad169d4d44974526ae97f28c59 2023-04-28 08:53:49,100 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/81422f1c28ff48088bb88343cabf36db as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/81422f1c28ff48088bb88343cabf36db 2023-04-28 08:53:49,108 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/81422f1c28ff48088bb88343cabf36db, entries=150, sequenceid=216, filesize=11.8 K 2023-04-28 08:53:49,110 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/809bd19ab67c4d4b8b626f4bc84e0eab as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/809bd19ab67c4d4b8b626f4bc84e0eab 2023-04-28 08:53:49,117 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/809bd19ab67c4d4b8b626f4bc84e0eab, entries=150, sequenceid=216, filesize=11.8 K 2023-04-28 08:53:49,119 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/8efed1ad169d4d44974526ae97f28c59 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8efed1ad169d4d44974526ae97f28c59 2023-04-28 08:53:49,133 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8efed1ad169d4d44974526ae97f28c59, entries=150, sequenceid=216, filesize=11.8 K 2023-04-28 08:53:49,139 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6dfeafaee81d2f031bddf09740078796 in 153ms, sequenceid=216, compaction requested=true 2023-04-28 08:53:49,139 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:49,139 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,139 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:49,140 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:49,140 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,140 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:49,140 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:49,142 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,143 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:49,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dc6caa8[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:49,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:49,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:49,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:49,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:49,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:49,154 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:49,154 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,154 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:49,155 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:49,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:49,160 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:49,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:49,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:49,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:49,160 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:49,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,169 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:49,171 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 08:53:49,171 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:49,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:49,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:49,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:49,172 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:49,172 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:49,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,174 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:49,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:49,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,179 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:49,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:49,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/9e3471d473e94ad4beb75f0a5ff045f2 2023-04-28 08:53:49,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672089209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672089211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672089212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672089212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672089215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:49,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:49,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:49,287 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:49,287 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,288 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:49,288 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:49,288 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,288 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,290 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,291 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:49,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:49,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:49,292 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:49,292 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:49,293 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,293 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:49,293 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:49,293 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:49,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:49,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:49,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c46cd4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:49,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:49,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:49,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:49,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:49,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:49,303 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:49,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:49,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:49,312 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:49,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:49,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:49,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:49,312 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:49,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,320 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,321 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:49,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:49,324 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:49,325 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:49,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672089324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672089325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672089325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672089326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672089326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,425 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:49,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:49,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:49,430 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:49,430 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:49,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:49,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:49,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:49,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:49,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:49,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:49,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:49,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:49,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:49,452 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:49,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:49,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:49,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:49,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:49,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:49,459 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3925e9f2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:49,462 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:49,462 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:49,464 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,464 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:49,465 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:49,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:49,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:49,473 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:49,473 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:49,473 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:49,473 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:49,473 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:49,475 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,476 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:49,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:49,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:49,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672089531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672089532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672089532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672089532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672089533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:49,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:49,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:49,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:49,592 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:49,593 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,593 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:49,593 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:49,593 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,593 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,595 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,595 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,595 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,595 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:49,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:49,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:49,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:49,596 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:49,596 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,596 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:49,600 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,600 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,600 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,600 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@372dfd6c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:49,602 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:49,602 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,604 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,604 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:49,604 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:49,604 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:49,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:49,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:49,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:49,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:49,610 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:49,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:49,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:49,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:49,610 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:49,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:49,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,612 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:49,612 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,614 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:49,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:49,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:49,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:49,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:49,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,620 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:49,620 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:49,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:49,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:49,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/e38b94edde5f4570b6fa7289eb535253 2023-04-28 08:53:49,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/da22208af7bb4f7991bc0f4656909401 2023-04-28 08:53:49,689 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/9e3471d473e94ad4beb75f0a5ff045f2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/9e3471d473e94ad4beb75f0a5ff045f2 2023-04-28 08:53:49,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/9e3471d473e94ad4beb75f0a5ff045f2, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:53:49,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/e38b94edde5f4570b6fa7289eb535253 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e38b94edde5f4570b6fa7289eb535253 2023-04-28 08:53:49,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e38b94edde5f4570b6fa7289eb535253, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:53:49,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/da22208af7bb4f7991bc0f4656909401 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/da22208af7bb4f7991bc0f4656909401 2023-04-28 08:53:49,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/da22208af7bb4f7991bc0f4656909401, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:53:49,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 6dfeafaee81d2f031bddf09740078796 in 548ms, sequenceid=230, compaction requested=true 2023-04-28 08:53:49,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:49,720 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:49,720 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:53:49,720 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:53:49,720 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:49,720 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:49,720 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:49,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:49,723 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60318 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:53:49,723 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:49,723 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,723 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/5245b57038024a18bfe60af8450b59df, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/75730495d9544463a37fd901e88d204d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0fed88a0f2074303913b959a0d301671, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/809bd19ab67c4d4b8b626f4bc84e0eab, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e38b94edde5f4570b6fa7289eb535253] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=58.9 K 2023-04-28 08:53:49,724 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60318 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:53:49,724 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5245b57038024a18bfe60af8450b59df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682672025816 2023-04-28 08:53:49,725 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:49,725 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,725 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/17756b75d65840a794a7a45451fabf4a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4ee24e314e2e438390ff91df14a92bf2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f88e2c5e0f014fb2b1d6c8b75c76e1de, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/81422f1c28ff48088bb88343cabf36db, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/9e3471d473e94ad4beb75f0a5ff045f2] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=58.9 K 2023-04-28 08:53:49,726 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 75730495d9544463a37fd901e88d204d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682672026009 2023-04-28 08:53:49,726 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 17756b75d65840a794a7a45451fabf4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682672025816 2023-04-28 08:53:49,727 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0fed88a0f2074303913b959a0d301671, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682672027199 2023-04-28 08:53:49,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:49,728 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4ee24e314e2e438390ff91df14a92bf2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682672026009 2023-04-28 08:53:49,728 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 809bd19ab67c4d4b8b626f4bc84e0eab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682672027881 2023-04-28 08:53:49,729 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f88e2c5e0f014fb2b1d6c8b75c76e1de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682672027199 2023-04-28 08:53:49,729 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 81422f1c28ff48088bb88343cabf36db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682672027881 2023-04-28 08:53:49,730 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9e3471d473e94ad4beb75f0a5ff045f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672029041 2023-04-28 08:53:49,731 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e38b94edde5f4570b6fa7289eb535253, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672029041 2023-04-28 08:53:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:49,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:49,731 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:49,731 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,731 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:49,731 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:49,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,739 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,740 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:49,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:49,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:49,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:49,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:49,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:49,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:49,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:49,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:49,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:49,750 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,750 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:49,750 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 08:53:49,750 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:49,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:49,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:49,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:49,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:49,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:49,759 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#27 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:49,766 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#28 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:49,821 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/04aa814dcfe040079ee16e5ed3f50867 2023-04-28 08:53:49,831 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/756a5ce882b248adaae91ca51c0b1a54 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/756a5ce882b248adaae91ca51c0b1a54 2023-04-28 08:53:49,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:49,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:49,842 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into 756a5ce882b248adaae91ca51c0b1a54(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:49,842 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:49,842 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=11, startTime=1682672029720; duration=0sec 2023-04-28 08:53:49,843 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:49,843 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:53:49,849 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60318 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:53:49,849 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:53:49,849 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:49,849 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8b35c53f1c2e43fb9d96e3673b1a1a51, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/2d5faae90a6343d9ba8bfb92876399be, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1a3b4362c4324c1db12393773f8d3bb9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8efed1ad169d4d44974526ae97f28c59, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/da22208af7bb4f7991bc0f4656909401] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=58.9 K 2023-04-28 08:53:49,850 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/063357517e71458cb5b24654d41aac4f 2023-04-28 08:53:49,851 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8b35c53f1c2e43fb9d96e3673b1a1a51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1682672025816 2023-04-28 08:53:49,856 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 2d5faae90a6343d9ba8bfb92876399be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682672026009 2023-04-28 08:53:49,857 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1a3b4362c4324c1db12393773f8d3bb9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1682672027199 2023-04-28 08:53:49,858 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8efed1ad169d4d44974526ae97f28c59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=216, earliestPutTs=1682672027881 2023-04-28 08:53:49,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672089846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672089848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672089858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,862 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting da22208af7bb4f7991bc0f4656909401, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672029041 2023-04-28 08:53:49,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672089859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672089861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,889 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#32 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:49,905 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/df9223f36f644aeb860dbba365f5bb4b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/df9223f36f644aeb860dbba365f5bb4b 2023-04-28 08:53:49,914 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into df9223f36f644aeb860dbba365f5bb4b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:49,914 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:49,914 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=11, startTime=1682672029720; duration=0sec 2023-04-28 08:53:49,915 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:49,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672089962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672089963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672089963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:49,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672089967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:49,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672089967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672090166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672090167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672090167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672090171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672090173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,226 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/7fdf66c6820841ec8d936b623b3409db as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/7fdf66c6820841ec8d936b623b3409db 2023-04-28 08:53:50,235 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into 7fdf66c6820841ec8d936b623b3409db(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:50,235 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:50,235 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=11, startTime=1682672029720; duration=0sec 2023-04-28 08:53:50,236 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:50,292 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/0dd98b8f3a64455ea80838a55dfa1e60 2023-04-28 08:53:50,304 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/04aa814dcfe040079ee16e5ed3f50867 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/04aa814dcfe040079ee16e5ed3f50867 2023-04-28 08:53:50,312 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/04aa814dcfe040079ee16e5ed3f50867, entries=150, sequenceid=249, filesize=11.8 K 2023-04-28 08:53:50,318 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/063357517e71458cb5b24654d41aac4f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/063357517e71458cb5b24654d41aac4f 2023-04-28 08:53:50,325 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/063357517e71458cb5b24654d41aac4f, entries=150, sequenceid=249, filesize=11.8 K 2023-04-28 08:53:50,327 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/0dd98b8f3a64455ea80838a55dfa1e60 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0dd98b8f3a64455ea80838a55dfa1e60 2023-04-28 08:53:50,335 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0dd98b8f3a64455ea80838a55dfa1e60, entries=150, sequenceid=249, filesize=11.8 K 2023-04-28 08:53:50,338 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6dfeafaee81d2f031bddf09740078796 in 588ms, sequenceid=249, compaction requested=false 2023-04-28 08:53:50,338 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:50,338 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,338 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:50,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:50,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:50,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:50,341 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c45e2ab[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:50,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:50,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,349 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:50,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:50,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:50,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:50,351 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,352 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,352 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,352 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:50,354 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:50,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:50,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:50,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:50,354 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:50,357 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,357 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,357 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,359 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,361 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,365 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,365 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,365 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,366 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,366 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,366 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,366 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,366 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,365 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:50,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:50,366 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:50,366 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:50,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,466 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:50,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:50,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:50,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 08:53:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:50,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:50,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:50,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:50,476 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:50,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:50,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:50,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,480 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:50,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:50,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:50,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:50,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:50,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:50,493 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,493 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:50,494 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:50,494 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/7aed17c7eb3a4cda824a5652d4436199 2023-04-28 08:53:50,494 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:50,494 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:50,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:50,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:50,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:50,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672090487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,496 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672090489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672090489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672090490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672090495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:50,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2982953b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:50,505 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:50,505 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,508 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:50,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:50,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:50,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:50,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:50,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:50,519 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:50,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:50,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:50,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:50,519 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:50,521 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,521 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,521 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/0d89d62784914232ba5e322ff334e98f 2023-04-28 08:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,527 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,537 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:50,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,537 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:50,538 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:50,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672090596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672090604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672090604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672090604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672090606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,638 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:50,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:50,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:50,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:50,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:50,642 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:50,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:50,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:50,643 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,643 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:50,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:50,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:50,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:50,649 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:50,649 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,649 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:50,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:50,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:50,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:50,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:50,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:50,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:50,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:50,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:50,662 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,664 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,664 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:50,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:50,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffe9380[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:50,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:50,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,671 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:50,671 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:50,671 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:50,673 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,673 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:50,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,675 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:50,675 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,675 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:50,680 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:50,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:50,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:50,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:50,680 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:50,683 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,683 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,683 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,722 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,722 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,722 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:50,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:50,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,722 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,723 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,723 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,723 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,723 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,723 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,723 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:50,724 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:50,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672090800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672090806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672090808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672090810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:50,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672090811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,824 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:50,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:50,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:50,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:50,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:50,829 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:50,829 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:50,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:50,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,832 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,832 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,832 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,833 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:50,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:50,834 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,834 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:50,834 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:50,834 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:50,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:50,837 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,837 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,838 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,838 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6760df8e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:50,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:50,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:50,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:50,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:50,841 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,841 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:50,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:50,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:50,849 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:50,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:50,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:50,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:50,849 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:50,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,852 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:50,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:50,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:50,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:50,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:50,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:50,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:50,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:50,962 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:50,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:50,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:50,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/5c8b3bb8106f442eac3ed0bc8fb04f24 2023-04-28 08:53:50,965 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:50,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:50,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:50,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:50,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:50,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:50,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:50,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:50,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:50,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:50,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:50,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:50,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:50,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:50,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:50,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:50,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:50,974 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,974 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,976 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,976 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d32973[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:50,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:50,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,981 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/7aed17c7eb3a4cda824a5652d4436199 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7aed17c7eb3a4cda824a5652d4436199 2023-04-28 08:53:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:50,981 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:50,981 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:50,981 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:50,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:50,984 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:50,984 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:50,984 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:50,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:50,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:50,995 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:50,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:50,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:50,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:50,995 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:50,998 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,998 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:50,998 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:50,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:50,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:50,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7aed17c7eb3a4cda824a5652d4436199, entries=200, sequenceid=267, filesize=14.3 K 2023-04-28 08:53:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:50,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/0d89d62784914232ba5e322ff334e98f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0d89d62784914232ba5e322ff334e98f 2023-04-28 08:53:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,009 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0d89d62784914232ba5e322ff334e98f, entries=150, sequenceid=267, filesize=11.9 K 2023-04-28 08:53:51,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/5c8b3bb8106f442eac3ed0bc8fb04f24 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/5c8b3bb8106f442eac3ed0bc8fb04f24 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:51,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,021 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,021 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:51,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,022 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:51,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/5c8b3bb8106f442eac3ed0bc8fb04f24, entries=150, sequenceid=267, filesize=11.9 K 2023-04-28 08:53:51,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6dfeafaee81d2f031bddf09740078796 in 561ms, sequenceid=267, compaction requested=true 2023-04-28 08:53:51,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:51,034 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:51,034 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:51,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:51,034 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:51,035 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39006 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:51,035 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:51,035 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,035 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36566 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:51,035 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/756a5ce882b248adaae91ca51c0b1a54, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/04aa814dcfe040079ee16e5ed3f50867, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7aed17c7eb3a4cda824a5652d4436199] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=38.1 K 2023-04-28 08:53:51,035 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:51,036 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,036 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/7fdf66c6820841ec8d936b623b3409db, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/063357517e71458cb5b24654d41aac4f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0d89d62784914232ba5e322ff334e98f] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=35.7 K 2023-04-28 08:53:51,036 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 756a5ce882b248adaae91ca51c0b1a54, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672029041 2023-04-28 08:53:51,036 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7fdf66c6820841ec8d936b623b3409db, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672029041 2023-04-28 08:53:51,041 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 04aa814dcfe040079ee16e5ed3f50867, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682672029202 2023-04-28 08:53:51,041 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 063357517e71458cb5b24654d41aac4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682672029202 2023-04-28 08:53:51,041 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0d89d62784914232ba5e322ff334e98f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682672029845 2023-04-28 08:53:51,042 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7aed17c7eb3a4cda824a5652d4436199, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682672029845 2023-04-28 08:53:51,054 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:51,056 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#37 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:51,074 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/c5b4f8e356554912af433636d1c4995c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c5b4f8e356554912af433636d1c4995c 2023-04-28 08:53:51,081 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/56c49e996f2d4571aa5a11af61022ea1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/56c49e996f2d4571aa5a11af61022ea1 2023-04-28 08:53:51,084 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into c5b4f8e356554912af433636d1c4995c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:51,084 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,084 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=13, startTime=1682672031034; duration=0sec 2023-04-28 08:53:51,084 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:51,084 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:51,091 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36566 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:51,091 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:53:51,091 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,091 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/df9223f36f644aeb860dbba365f5bb4b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0dd98b8f3a64455ea80838a55dfa1e60, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/5c8b3bb8106f442eac3ed0bc8fb04f24] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=35.7 K 2023-04-28 08:53:51,092 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting df9223f36f644aeb860dbba365f5bb4b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672029041 2023-04-28 08:53:51,093 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0dd98b8f3a64455ea80838a55dfa1e60, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1682672029202 2023-04-28 08:53:51,094 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5c8b3bb8106f442eac3ed0bc8fb04f24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682672029845 2023-04-28 08:53:51,099 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into 56c49e996f2d4571aa5a11af61022ea1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:51,101 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,101 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=13, startTime=1682672031034; duration=0sec 2023-04-28 08:53:51,101 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:51,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:51,109 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 08:53:51,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:51,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,111 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#38 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:51,122 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:51,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:51,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:51,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:51,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:51,132 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:51,132 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,132 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:51,132 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:51,133 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,133 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,134 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/6984aaf45cf34f5580f2c4d326a06558 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6984aaf45cf34f5580f2c4d326a06558 2023-04-28 08:53:51,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/1b5487446c5246b6b2d5f74c1d864d28 2023-04-28 08:53:51,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672091132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672091137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672091138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672091141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,148 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into 6984aaf45cf34f5580f2c4d326a06558(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:51,148 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,148 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=13, startTime=1682672031034; duration=0sec 2023-04-28 08:53:51,148 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:51,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:51,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672091152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,157 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,157 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:51,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:51,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:51,165 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:51,165 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,166 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:51,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:51,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:51,171 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,171 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:51,171 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:51,171 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:51,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:51,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:51,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:51,176 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,176 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:51,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39454f9f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:51,180 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:51,180 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,182 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:51,182 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:51,182 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:51,182 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,182 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:51,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/2bdf7083765843fcbd7f9baf73d60403 2023-04-28 08:53:51,184 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,184 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,184 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,184 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:51,189 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:51,189 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:51,189 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:51,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:51,190 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:51,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,195 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,195 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,196 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/85b8fbffe0354c3282b5a0e00919c369 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:51,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:51,211 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:51,211 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:51,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/1b5487446c5246b6b2d5f74c1d864d28 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1b5487446c5246b6b2d5f74c1d864d28 2023-04-28 08:53:51,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1b5487446c5246b6b2d5f74c1d864d28, entries=200, sequenceid=287, filesize=14.3 K 2023-04-28 08:53:51,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/2bdf7083765843fcbd7f9baf73d60403 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/2bdf7083765843fcbd7f9baf73d60403 2023-04-28 08:53:51,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/2bdf7083765843fcbd7f9baf73d60403, entries=150, sequenceid=287, filesize=11.9 K 2023-04-28 08:53:51,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/85b8fbffe0354c3282b5a0e00919c369 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/85b8fbffe0354c3282b5a0e00919c369 2023-04-28 08:53:51,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/85b8fbffe0354c3282b5a0e00919c369, entries=150, sequenceid=287, filesize=11.9 K 2023-04-28 08:53:51,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 6dfeafaee81d2f031bddf09740078796 in 132ms, sequenceid=287, compaction requested=false 2023-04-28 08:53:51,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:51,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:53:51,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:51,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:51,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:51,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672091269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672091270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672091272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672091277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672091277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,311 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:51,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:51,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:51,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:51,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:51,323 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:51,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:51,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:51,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,326 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,326 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:51,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:51,335 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:51,335 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:51,335 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:51,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:51,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:51,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:51,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:51,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:51,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:51,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:51,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:51,341 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:51,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726ca9a4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:51,344 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:51,344 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,347 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,347 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:51,347 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:51,347 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:51,347 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:51,347 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:51,348 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:51,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:51,352 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:51,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:51,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:51,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:51,352 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:51,354 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,354 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,354 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,356 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:51,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,359 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,359 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:51,359 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:51,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672091379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672091379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672091380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672091383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672091384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:51,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:51,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:51,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:51,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:51,467 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:51,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:51,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:51,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,473 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,474 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,474 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,474 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:51,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:51,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:51,480 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:51,480 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,480 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:51,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:51,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:51,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:51,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,485 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:51,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@131184c8[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:51,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:51,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,489 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:51,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:51,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:51,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:51,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:51,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:51,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:51,491 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:51,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:51,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:51,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:51,491 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:51,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,497 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:51,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:51,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,500 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:51,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,501 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,501 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,501 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:51,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672091583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672091583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672091583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672091587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672091589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,601 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:51,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:51,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:51,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:51,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:51,606 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:51,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:51,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:51,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,608 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,608 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:51,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:51,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:51,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:51,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:51,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:51,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:51,614 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,614 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:51,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:51,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:51,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:51,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:51,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:51,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e14af68[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:51,618 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:51,618 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:51,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:51,621 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:51,621 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:51,622 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:51,626 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:51,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:51,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:51,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:51,626 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:51,627 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,627 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,627 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,633 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:51,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:51,636 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,635 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,636 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:51,636 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:51,636 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,636 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,636 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,636 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/fa0ad527748b41a489dd3b423cc3da48 2023-04-28 08:53:51,726 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/ce8e2213258b48538bb13b3881044593 2023-04-28 08:53:51,736 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:51,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:51,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:51,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:51,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:51,752 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:51,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:51,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:51,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:51,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:51,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:51,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:51,778 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:51,778 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:51,778 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,778 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:51,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:51,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:51,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:51,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:51,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:51,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:51,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/bbb9435333094166b0aad8208929bb51 2023-04-28 08:53:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:51,791 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:51,791 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dbe3082[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:51,794 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/fa0ad527748b41a489dd3b423cc3da48 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/fa0ad527748b41a489dd3b423cc3da48 2023-04-28 08:53:51,795 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,795 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:51,795 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:51,796 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:51,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:51,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:51,798 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/fa0ad527748b41a489dd3b423cc3da48, entries=200, sequenceid=304, filesize=14.3 K 2023-04-28 08:53:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/ce8e2213258b48538bb13b3881044593 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ce8e2213258b48538bb13b3881044593 2023-04-28 08:53:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:51,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:51,803 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:51,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:51,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:51,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:51,803 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:51,805 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,806 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,806 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,812 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,815 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:51,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,816 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:51,816 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:51,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ce8e2213258b48538bb13b3881044593, entries=150, sequenceid=304, filesize=11.9 K 2023-04-28 08:53:51,830 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/bbb9435333094166b0aad8208929bb51 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/bbb9435333094166b0aad8208929bb51 2023-04-28 08:53:51,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/bbb9435333094166b0aad8208929bb51, entries=150, sequenceid=304, filesize=11.9 K 2023-04-28 08:53:51,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dfeafaee81d2f031bddf09740078796 in 594ms, sequenceid=304, compaction requested=true 2023-04-28 08:53:51,838 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:51,838 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:51,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:51,838 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:51,839 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:51,840 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41848 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:51,840 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:51,840 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,840 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:51,840 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/56c49e996f2d4571aa5a11af61022ea1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1b5487446c5246b6b2d5f74c1d864d28, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/fa0ad527748b41a489dd3b423cc3da48] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=40.9 K 2023-04-28 08:53:51,840 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:51,840 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,840 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c5b4f8e356554912af433636d1c4995c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/2bdf7083765843fcbd7f9baf73d60403, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ce8e2213258b48538bb13b3881044593] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.1 K 2023-04-28 08:53:51,840 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 56c49e996f2d4571aa5a11af61022ea1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682672029845 2023-04-28 08:53:51,842 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1b5487446c5246b6b2d5f74c1d864d28, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682672030487 2023-04-28 08:53:51,842 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c5b4f8e356554912af433636d1c4995c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682672029845 2023-04-28 08:53:51,844 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting fa0ad527748b41a489dd3b423cc3da48, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672031128 2023-04-28 08:53:51,844 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2bdf7083765843fcbd7f9baf73d60403, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682672030488 2023-04-28 08:53:51,845 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ce8e2213258b48538bb13b3881044593, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672031136 2023-04-28 08:53:51,859 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:51,872 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:51,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:51,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 08:53:51,902 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/aed6111d1b684bfeb86e0ba39ff45af4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aed6111d1b684bfeb86e0ba39ff45af4 2023-04-28 08:53:51,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:51,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:51,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:51,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:51,913 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/bf30db0a5f784bcc9d85ea3de7c5edad as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bf30db0a5f784bcc9d85ea3de7c5edad 2023-04-28 08:53:51,916 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:51,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:51,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:51,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:51,923 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:51,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:51,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:51,924 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,924 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:51,928 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into aed6111d1b684bfeb86e0ba39ff45af4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:51,928 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,928 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=13, startTime=1682672031838; duration=0sec 2023-04-28 08:53:51,929 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:51,929 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:51,930 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:51,930 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:53:51,931 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,931 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,931 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6984aaf45cf34f5580f2c4d326a06558, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/85b8fbffe0354c3282b5a0e00919c369, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/bbb9435333094166b0aad8208929bb51] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.1 K 2023-04-28 08:53:51,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,932 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 6984aaf45cf34f5580f2c4d326a06558, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1682672029845 2023-04-28 08:53:51,933 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into bf30db0a5f784bcc9d85ea3de7c5edad(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:51,958 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:51,958 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=13, startTime=1682672031838; duration=0sec 2023-04-28 08:53:51,958 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:51,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,961 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 85b8fbffe0354c3282b5a0e00919c369, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1682672030488 2023-04-28 08:53:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:51,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672091927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:51,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:51,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672091928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:51,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:51,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:51,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:51,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:51,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:51,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:51,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:51,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672091929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,964 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bbb9435333094166b0aad8208929bb51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672031136 2023-04-28 08:53:51,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:51,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:51,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:51,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:51,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d 2023-04-28 08:53:51,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672091932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:51,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672091932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,971 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,972 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,972 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:51,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a089b18[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:51,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:51,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:51,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:51,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:51,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:51,982 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,982 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:51,982 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,982 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:51,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:51,987 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:51,987 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:51,987 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:51,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:51,988 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:51,990 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:51,990 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,992 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:51,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:51,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:51,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:51,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:51,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:51,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:51,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:51,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:51,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,000 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,000 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:52,019 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/6122f45e3b3a42ed9d573b045fd80b20 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6122f45e3b3a42ed9d573b045fd80b20 2023-04-28 08:53:52,025 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into 6122f45e3b3a42ed9d573b045fd80b20(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:52,025 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:52,026 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=13, startTime=1682672031838; duration=0sec 2023-04-28 08:53:52,026 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:52,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672092064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672092065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672092066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672092070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672092071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,100 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:52,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:52,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:52,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:52,104 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:52,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:52,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:52,105 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,105 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,106 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,106 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,107 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,107 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:52,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:52,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:52,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:52,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:52,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:52,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:52,109 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:52,109 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,109 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:52,109 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:52,109 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,109 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:52,109 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:52,109 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,109 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:52,109 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:52,110 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,111 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,111 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,111 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15a07955[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:52,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:52,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,115 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:52,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:52,115 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,115 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:52,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:52,118 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:52,118 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,118 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:52,118 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:52,121 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:52,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:52,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:52,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:52,122 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:52,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,126 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,130 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,130 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,130 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,130 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,130 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:52,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:52,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,130 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,130 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,131 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,131 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:52,131 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,131 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:52,131 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,231 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:52,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:52,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:52,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:52,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:52,235 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:52,235 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,235 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:52,235 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:52,235 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,236 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,237 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,237 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,238 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,238 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:52,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:52,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:52,239 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:52,239 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,239 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:52,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:52,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:52,240 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,240 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:52,240 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:52,240 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:52,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:52,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:52,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:52,243 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:52,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ee4fb66[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:52,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:52,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:52,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:52,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:52,247 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,247 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:52,256 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:52,259 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:52,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:52,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:52,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:52,259 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:52,264 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,264 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,264 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,267 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672092267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672092267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672092268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:52,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:52,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:52,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,271 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:52,271 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:52,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59018 deadline: 1682672092274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59076 deadline: 1682672092274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,371 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:52,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:52,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:52,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:52,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:52,376 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:52,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:52,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:52,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,378 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:52,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:52,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:52,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:52,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:52,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:52,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:52,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:52,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:52,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:52,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:52,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:52,382 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:52,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5232e618[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:52,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:52,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,386 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:52,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:52,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:52,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:52,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:52,391 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:52,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:52,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:52,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:52,391 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:52,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,393 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:52,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,399 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:52,400 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:52,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/9305c385897d41719d8aa1a0c5f61bf1 2023-04-28 08:53:52,500 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:52,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:52,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:52,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:52,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:52,504 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:52,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:52,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:52,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:52,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:52,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:52,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:52,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:52,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:52,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2c771d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:52,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:52,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,515 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,515 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:52,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:52,515 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:52,515 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:52,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:52,517 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:52,519 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:52,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:52,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:52,520 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:52,520 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:52,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,528 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,528 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,528 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,528 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:52,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:52,529 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:52,529 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:52,529 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:52,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672092572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59032 deadline: 1682672092573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59024 deadline: 1682672092573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672092578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:52,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672092583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,630 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:52,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:52,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:52,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:52,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:52,642 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:52,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:52,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:52,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,643 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,645 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:52,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:52,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:52,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:52,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:52,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:52,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:52,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:52,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:52,648 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:52,648 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:52,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:52,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:52,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:52,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,651 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:52,651 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a50cf1c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:52,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:52,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,657 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:52,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:52,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:52,658 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,658 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:52,659 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:52,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:52,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:52,664 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:52,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:52,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:52,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:52,664 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:52,666 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,666 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,666 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,667 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,669 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:52,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:52,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,678 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,678 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,778 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:52,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:52,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:52,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:52,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:52,783 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:52,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:52,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:52,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,785 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:52,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:52,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:52,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:52,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:52,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:52,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:52,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:52,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:52,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:52,790 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,791 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,791 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:52,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13763afb[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:52,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:52,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,798 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:52,797 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:52,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:52,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:52,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:52,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:52,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:52,802 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:52,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:52,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:52,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:52,803 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:52,805 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,805 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,806 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:52,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:52,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:52,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:52,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:52,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:52,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:52,814 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:52,814 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:52,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=322 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/9b4196688cfd4904b1c81322edfe0417 2023-04-28 08:53:52,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d 2023-04-28 08:53:52,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d, entries=150, sequenceid=322, filesize=11.9 K 2023-04-28 08:53:52,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/9305c385897d41719d8aa1a0c5f61bf1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9305c385897d41719d8aa1a0c5f61bf1 2023-04-28 08:53:52,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9305c385897d41719d8aa1a0c5f61bf1, entries=150, sequenceid=322, filesize=11.9 K 2023-04-28 08:53:52,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/9b4196688cfd4904b1c81322edfe0417 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9b4196688cfd4904b1c81322edfe0417 2023-04-28 08:53:52,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9b4196688cfd4904b1c81322edfe0417, entries=150, sequenceid=322, filesize=11.9 K 2023-04-28 08:53:52,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 6dfeafaee81d2f031bddf09740078796 in 981ms, sequenceid=322, compaction requested=false 2023-04-28 08:53:52,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:52,914 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:52,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:52,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:52,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:52,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:52,918 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:52,918 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,918 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:52,918 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:52,919 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,919 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,921 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:52,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:52,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:52,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:52,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:52,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:52,923 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:52,923 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:52,923 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:52,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:52,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:52,927 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:52,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:53,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:53,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:53,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59062 deadline: 1682672093095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672093097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672093098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672093098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672093099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672093201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672093202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672093204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672093204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672093205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,341 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/50f16544715842d38f9c37ef4e8811a7 2023-04-28 08:53:53,354 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/bced7928de3442fc82a9dcae455393c6 2023-04-28 08:53:53,373 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/baf672379c864b77b0a1e9d1552ee2d2 2023-04-28 08:53:53,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/50f16544715842d38f9c37ef4e8811a7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/50f16544715842d38f9c37ef4e8811a7 2023-04-28 08:53:53,399 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/50f16544715842d38f9c37ef4e8811a7, entries=150, sequenceid=340, filesize=11.9 K 2023-04-28 08:53:53,401 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/bced7928de3442fc82a9dcae455393c6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bced7928de3442fc82a9dcae455393c6 2023-04-28 08:53:53,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672093407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672093407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672093410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,413 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bced7928de3442fc82a9dcae455393c6, entries=150, sequenceid=340, filesize=11.9 K 2023-04-28 08:53:53,414 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/baf672379c864b77b0a1e9d1552ee2d2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/baf672379c864b77b0a1e9d1552ee2d2 2023-04-28 08:53:53,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672093407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672093411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,423 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/baf672379c864b77b0a1e9d1552ee2d2, entries=150, sequenceid=340, filesize=11.9 K 2023-04-28 08:53:53,425 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 6dfeafaee81d2f031bddf09740078796 in 498ms, sequenceid=340, compaction requested=true 2023-04-28 08:53:53,425 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:53,425 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:53,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:53,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:53,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:53,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:53,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:53,429 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:53,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:53,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:53,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:53,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56311cab[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:53,433 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:53,433 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:53,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:53,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:53,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:53,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:53,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:53,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:53,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:53,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:53,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:53,447 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:53,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:53,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:53,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:53,447 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:53,450 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,450 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:53,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,450 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:53,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:53,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:53,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:53,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:53,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:53,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:53,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:53,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:53,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:53,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:53,458 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:53,458 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:53,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:53,558 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:53,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:53,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:53,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:53,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:53,562 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:53,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:53,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:53,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:53,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:53,564 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:53,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:53,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:53,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:53,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:53,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:53,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:53,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:53,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:53,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:53,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:53,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:53,566 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:53,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:53,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:53,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:53,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672093765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672093766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672093767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672093768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672093770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672093871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672093871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672093871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:53,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672093872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672093872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:53,978 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/26fa5325d1b54e01ba9bba41e6d6448a 2023-04-28 08:53:54,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672094075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672094076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672094077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672094077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672094078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672094383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672094383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672094385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672094385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672094387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,400 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/1fa14733c41a403ca52b1c8458c3284d 2023-04-28 08:53:54,814 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c18435c8410a4ec481d897ba7ecfaf0c 2023-04-28 08:53:54,821 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/26fa5325d1b54e01ba9bba41e6d6448a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/26fa5325d1b54e01ba9bba41e6d6448a 2023-04-28 08:53:54,832 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/26fa5325d1b54e01ba9bba41e6d6448a, entries=150, sequenceid=358, filesize=11.9 K 2023-04-28 08:53:54,834 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/1fa14733c41a403ca52b1c8458c3284d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/1fa14733c41a403ca52b1c8458c3284d 2023-04-28 08:53:54,840 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/1fa14733c41a403ca52b1c8458c3284d, entries=150, sequenceid=358, filesize=11.9 K 2023-04-28 08:53:54,841 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c18435c8410a4ec481d897ba7ecfaf0c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c18435c8410a4ec481d897ba7ecfaf0c 2023-04-28 08:53:54,848 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c18435c8410a4ec481d897ba7ecfaf0c, entries=150, sequenceid=358, filesize=11.9 K 2023-04-28 08:53:54,849 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6dfeafaee81d2f031bddf09740078796 in 1283ms, sequenceid=358, compaction requested=true 2023-04-28 08:53:54,850 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:54,850 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:54,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:54,852 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:54,852 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,852 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:54,852 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:54,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:54,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29da1def[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:54,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:54,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,858 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:54,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:54,858 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,858 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:54,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:54,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:54,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:54,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:54,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:54,868 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:54,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:54,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:54,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:54,869 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:54,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:54,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:54,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:54,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,876 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,884 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:54,884 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:54,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:54,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:54,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:54,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:54,884 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:54,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:54,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:54,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,885 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:54,885 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:54,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:54,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:54,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:54,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 08:53:54,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:54,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:54,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:54,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:54,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:54,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:54,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672094914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672094914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672094918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672094928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:54,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672094928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/63a38b82ffba41f9b2aaa2196b7a1e79 2023-04-28 08:53:54,985 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:54,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:54,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:54,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:54,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:54,995 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:54,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:54,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:54,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:54,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:54,995 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:54,997 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,997 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:54,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:54,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:54,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,003 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,003 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,003 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:55,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:55,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:55,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:55,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:55,011 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,012 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,012 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257e1fe7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:55,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:55,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:55,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:55,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:55,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:55,019 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:55,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:55,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:55,024 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:55,024 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:55,024 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:55,024 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:55,024 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:55,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,028 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,031 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,035 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:55,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:55,037 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:55,037 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:55,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672095029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672095030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672095029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672095035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672095036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,137 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:55,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:55,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:55,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:55,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:55,147 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:55,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:55,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:55,148 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,148 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,152 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,152 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,152 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:55,152 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:55,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:55,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:55,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:55,155 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,156 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d4851[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:55,158 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:55,158 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:55,162 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,162 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:55,162 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:55,162 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:55,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:55,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:55,168 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:55,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:55,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:55,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:55,168 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:55,171 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,171 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,171 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,172 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,174 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,181 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,181 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,181 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:55,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:55,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:55,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:55,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672095239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672095240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672095240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672095240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672095242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:55,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:55,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:55,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:55,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:55,285 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:55,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:55,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:55,286 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,286 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,287 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,287 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,288 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,288 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,289 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,289 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,290 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,290 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,290 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:55,290 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:55,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:55,292 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,292 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,292 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,292 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:55,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5977d851[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:55,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:55,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:55,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:55,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:55,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:55,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:55,304 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:55,304 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:55,304 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:55,304 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:55,304 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:55,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:55,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:55,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,315 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:55,316 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:55,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,316 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,316 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:55,383 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/ee73c7be46d7467a9c5c917ef946b970 2023-04-28 08:53:55,416 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:55,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:55,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:55,420 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:55,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:55,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:55,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,428 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,428 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,428 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:55,428 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:55,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:55,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:55,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:55,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:55,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20bb93f5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:55,432 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:55,432 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,433 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:55,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:55,434 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:55,434 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:55,435 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:55,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:55,439 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:55,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:55,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:55,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:55,439 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:55,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:55,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:55,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,450 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:55,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672095544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672095547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,550 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:55,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672095547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672095548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:55,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:55,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672095549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:55,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:55,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:55,553 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:55,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:55,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:55,554 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,554 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:55,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:55,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:55,559 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,560 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,560 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:55,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756f3122[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:55,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:55,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,563 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:55,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:55,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:55,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:55,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:55,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,565 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:55,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:55,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:55,568 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:55,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:55,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:55,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:55,569 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:55,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:55,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:55,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,577 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,577 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,576 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:55,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:55,577 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:55,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,677 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:55,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:55,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:55,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:55,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:55,681 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:55,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:55,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:55,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,685 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,685 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,685 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:55,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:55,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:55,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18bdc01e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:55,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:55,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,692 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:55,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:55,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:55,693 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,693 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:55,694 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:55,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:55,698 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:55,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:55,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:55,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:55,699 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:55,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,705 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,708 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,708 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:55,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:55,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:55,708 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,709 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:55,709 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,709 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,715 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,715 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,715 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,715 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,715 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,715 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,798 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/f7dfd1c1850e4978a906ba23b7bb982c 2023-04-28 08:53:55,810 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:55,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:55,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/63a38b82ffba41f9b2aaa2196b7a1e79 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/63a38b82ffba41f9b2aaa2196b7a1e79 2023-04-28 08:53:55,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:55,820 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:55,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:55,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:55,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,824 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,824 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,827 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,827 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,827 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,831 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,831 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,831 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:55,831 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,831 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/63a38b82ffba41f9b2aaa2196b7a1e79, entries=150, sequenceid=374, filesize=11.9 K 2023-04-28 08:53:55,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:55,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:55,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:55,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:55,832 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/ee73c7be46d7467a9c5c917ef946b970 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ee73c7be46d7467a9c5c917ef946b970 2023-04-28 08:53:55,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:55,833 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6f3608[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:55,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:55,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ee73c7be46d7467a9c5c917ef946b970, entries=150, sequenceid=374, filesize=11.9 K 2023-04-28 08:53:55,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/f7dfd1c1850e4978a906ba23b7bb982c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/f7dfd1c1850e4978a906ba23b7bb982c 2023-04-28 08:53:55,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:55,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:55,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:55,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:55,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:55,845 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:55,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:55,846 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:55,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:55,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:55,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:55,846 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:55,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/f7dfd1c1850e4978a906ba23b7bb982c, entries=150, sequenceid=374, filesize=11.9 K 2023-04-28 08:53:55,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dfeafaee81d2f031bddf09740078796 in 956ms, sequenceid=374, compaction requested=true 2023-04-28 08:53:55,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:55,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:55,848 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:53:55,849 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:53:55,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:55,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:55,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:55,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,851 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:53:55,851 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:55,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,851 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61442 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:53:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,851 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:55,851 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:55,851 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,851 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aed6111d1b684bfeb86e0ba39ff45af4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/50f16544715842d38f9c37ef4e8811a7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/26fa5325d1b54e01ba9bba41e6d6448a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/63a38b82ffba41f9b2aaa2196b7a1e79] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=60.0 K 2023-04-28 08:53:55,851 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bf30db0a5f784bcc9d85ea3de7c5edad, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9305c385897d41719d8aa1a0c5f61bf1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bced7928de3442fc82a9dcae455393c6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/1fa14733c41a403ca52b1c8458c3284d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ee73c7be46d7467a9c5c917ef946b970] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=60.0 K 2023-04-28 08:53:55,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,859 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting aed6111d1b684bfeb86e0ba39ff45af4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672031136 2023-04-28 08:53:55,859 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bf30db0a5f784bcc9d85ea3de7c5edad, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672031136 2023-04-28 08:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:55,859 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 5b4d1f5eb02d4dc4a88ee20aaeaec40d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682672031267 2023-04-28 08:53:55,860 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9305c385897d41719d8aa1a0c5f61bf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682672031267 2023-04-28 08:53:55,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:55,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,863 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 50f16544715842d38f9c37ef4e8811a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682672031926 2023-04-28 08:53:55,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:55,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:55,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,864 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:55,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:55,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:55,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:55,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,864 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:55,864 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:55,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:55,867 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 26fa5325d1b54e01ba9bba41e6d6448a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682672033092 2023-04-28 08:53:55,867 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bced7928de3442fc82a9dcae455393c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682672031926 2023-04-28 08:53:55,870 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 1fa14733c41a403ca52b1c8458c3284d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682672033092 2023-04-28 08:53:55,870 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 63a38b82ffba41f9b2aaa2196b7a1e79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672033767 2023-04-28 08:53:55,870 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ee73c7be46d7467a9c5c917ef946b970, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672033767 2023-04-28 08:53:55,889 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:55,891 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#61 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:55,965 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:55,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:55,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:55,971 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:55,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:55,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:55,972 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,972 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,975 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:55,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:55,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:55,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:55,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:55,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:55,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:55,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:55,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:55,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:55,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:55,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:55,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:55,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:55,983 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:53:55,984 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:55,984 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:55,984 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:55,984 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:55,984 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:55,984 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,001 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/4cb03e86981c4664b1221e82fd8a68d0 2023-04-28 08:53:56,034 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/a1ccf2c8ae0e45f6a8e64dc7da291b01 2023-04-28 08:53:56,054 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:56,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:56,055 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c75d0a6318924128afd70f9a7d439d84 2023-04-28 08:53:56,062 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/4cb03e86981c4664b1221e82fd8a68d0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4cb03e86981c4664b1221e82fd8a68d0 2023-04-28 08:53:56,073 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4cb03e86981c4664b1221e82fd8a68d0, entries=150, sequenceid=390, filesize=11.9 K 2023-04-28 08:53:56,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672096068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672096068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,076 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/a1ccf2c8ae0e45f6a8e64dc7da291b01 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a1ccf2c8ae0e45f6a8e64dc7da291b01 2023-04-28 08:53:56,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672096069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672096070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672096076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,085 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a1ccf2c8ae0e45f6a8e64dc7da291b01, entries=150, sequenceid=390, filesize=11.9 K 2023-04-28 08:53:56,086 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c75d0a6318924128afd70f9a7d439d84 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c75d0a6318924128afd70f9a7d439d84 2023-04-28 08:53:56,102 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c75d0a6318924128afd70f9a7d439d84, entries=150, sequenceid=390, filesize=11.9 K 2023-04-28 08:53:56,103 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dfeafaee81d2f031bddf09740078796 in 120ms, sequenceid=390, compaction requested=true 2023-04-28 08:53:56,103 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,103 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,103 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:56,103 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:56,103 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,103 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:56,103 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:56,105 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,105 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,106 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,106 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:56,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b7362b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:56,108 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:56,108 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,116 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:56,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:56,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:56,116 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,116 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:56,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:56,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,118 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:56,123 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:56,123 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:56,123 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:56,123 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:56,124 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:56,126 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,126 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,126 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,133 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,133 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:56,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:56,134 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,134 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:56,134 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:56,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:56,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 08:53:56,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:56,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:56,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:56,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/7a80ee07b40d4a829b10818d64018db1 2023-04-28 08:53:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672096195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672096200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672096201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672096202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672096202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/911ad2431aee42cdaa27f2c7618f3fdc 2023-04-28 08:53:56,234 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:56,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:56,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:56,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:56,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:56,240 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:56,240 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,240 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:56,240 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:56,241 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,241 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,243 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:56,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:56,245 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:56,245 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,245 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:56,245 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:56,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:56,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:56,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:56,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:56,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:56,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:56,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:56,248 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,248 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,248 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,248 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:56,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f5df15e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:56,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,255 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:56,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,255 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:56,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:56,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:56,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/e8a67cdc46734cc9a05dd40951fba8c2 2023-04-28 08:53:56,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:56,258 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:56,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:56,263 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:56,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:56,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:56,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:56,264 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:56,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,266 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,267 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,268 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/7a80ee07b40d4a829b10818d64018db1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7a80ee07b40d4a829b10818d64018db1 2023-04-28 08:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:56,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:56,273 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,273 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:56,273 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:56,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:56,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7a80ee07b40d4a829b10818d64018db1, entries=150, sequenceid=407, filesize=11.9 K 2023-04-28 08:53:56,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/911ad2431aee42cdaa27f2c7618f3fdc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/911ad2431aee42cdaa27f2c7618f3fdc 2023-04-28 08:53:56,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/911ad2431aee42cdaa27f2c7618f3fdc, entries=150, sequenceid=407, filesize=11.9 K 2023-04-28 08:53:56,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/e8a67cdc46734cc9a05dd40951fba8c2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e8a67cdc46734cc9a05dd40951fba8c2 2023-04-28 08:53:56,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e8a67cdc46734cc9a05dd40951fba8c2, entries=150, sequenceid=407, filesize=11.9 K 2023-04-28 08:53:56,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 6dfeafaee81d2f031bddf09740078796 in 122ms, sequenceid=407, compaction requested=true 2023-04-28 08:53:56,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 08:53:56,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-28 08:53:56,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-28 08:53:56,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:56,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:53:56,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:56,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,311 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:56,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:56,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,332 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/bd16023a68114b49b21d5fdac4e519bd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bd16023a68114b49b21d5fdac4e519bd 2023-04-28 08:53:56,337 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/3ee12e619990451b95d557bdb33c36ef as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/3ee12e619990451b95d557bdb33c36ef 2023-04-28 08:53:56,341 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into bd16023a68114b49b21d5fdac4e519bd(size=12.6 K), total size for store is 36.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:56,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,342 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,342 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=11, startTime=1682672035848; duration=0sec 2023-04-28 08:53:56,342 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-28 08:53:56,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672096334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,342 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-28 08:53:56,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672096338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672096338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,346 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into 3ee12e619990451b95d557bdb33c36ef(size=12.6 K), total size for store is 36.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:56,346 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,346 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=11, startTime=1682672035848; duration=0sec 2023-04-28 08:53:56,346 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-28 08:53:56,347 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85814 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-28 08:53:56,347 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:53:56,348 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,348 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6122f45e3b3a42ed9d573b045fd80b20, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9b4196688cfd4904b1c81322edfe0417, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/baf672379c864b77b0a1e9d1552ee2d2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c18435c8410a4ec481d897ba7ecfaf0c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/f7dfd1c1850e4978a906ba23b7bb982c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c75d0a6318924128afd70f9a7d439d84, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e8a67cdc46734cc9a05dd40951fba8c2] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=83.8 K 2023-04-28 08:53:56,348 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 7 compacting, 0 eligible, 16 blocking 2023-04-28 08:53:56,348 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:53:56,348 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:53:56,348 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. because compaction request was cancelled 2023-04-28 08:53:56,348 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:56,348 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 6122f45e3b3a42ed9d573b045fd80b20, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672031136 2023-04-28 08:53:56,349 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9b4196688cfd4904b1c81322edfe0417, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=322, earliestPutTs=1682672031267 2023-04-28 08:53:56,350 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting baf672379c864b77b0a1e9d1552ee2d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1682672031926 2023-04-28 08:53:56,352 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting c18435c8410a4ec481d897ba7ecfaf0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1682672033092 2023-04-28 08:53:56,352 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:56,352 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:56,352 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,352 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/3ee12e619990451b95d557bdb33c36ef, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a1ccf2c8ae0e45f6a8e64dc7da291b01, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/911ad2431aee42cdaa27f2c7618f3fdc] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.4 K 2023-04-28 08:53:56,352 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f7dfd1c1850e4978a906ba23b7bb982c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672033767 2023-04-28 08:53:56,353 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3ee12e619990451b95d557bdb33c36ef, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672033767 2023-04-28 08:53:56,353 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting c75d0a6318924128afd70f9a7d439d84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682672034915 2023-04-28 08:53:56,353 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e8a67cdc46734cc9a05dd40951fba8c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682672036063 2023-04-28 08:53:56,353 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a1ccf2c8ae0e45f6a8e64dc7da291b01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682672034915 2023-04-28 08:53:56,355 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 911ad2431aee42cdaa27f2c7618f3fdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682672036063 2023-04-28 08:53:56,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672096341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672096342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,373 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:56,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:56,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:56,379 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#69 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:56,392 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#70 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:56,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:56,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:56,395 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:56,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:56,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:56,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:56,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:56,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:56,430 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:56,430 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:56,431 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:56,431 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:56,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:56,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:56,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:56,433 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:56,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79967d9b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:56,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:56,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:56,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:56,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:56,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:56,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:56,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:56,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672096443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672096443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672096444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:56,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:56,449 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:56,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:56,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:56,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:56,449 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:56,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,455 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,462 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:56,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672096459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:56,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672096459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:56,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,463 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,463 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,563 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:56,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:56,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:56,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:56,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:56,566 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:56,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:56,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:56,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:56,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:56,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:56,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:56,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:56,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:56,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:56,570 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,570 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:56,570 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:56,570 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:56,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:56,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:56,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:56,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fcfdbd7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:56,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:56,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,578 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,579 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:56,579 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:56,579 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:56,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:56,580 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,580 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,580 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,580 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:56,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:56,585 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:56,585 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:56,585 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:56,585 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:56,585 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:56,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,589 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:56,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:56,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,595 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:56,595 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:56,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672096648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672096648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672096648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672096663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672096664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,695 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:56,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:56,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:56,699 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:56,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:56,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:56,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,701 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,702 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:56,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:56,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:56,704 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:56,704 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,704 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:56,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:56,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:56,705 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,705 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:56,705 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:56,705 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:56,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:56,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:56,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:56,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,707 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,707 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,707 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:56,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4573ac6b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:56,709 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:56,709 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,710 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,710 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:56,710 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:56,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:56,710 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:56,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:56,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:56,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:56,717 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:56,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:56,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:56,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:56,718 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:56,719 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,719 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,719 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,720 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:56,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,724 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,725 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:56,725 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:56,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,725 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/cdf38a1f28414673b582fa6759c83a42 2023-04-28 08:53:56,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/412d0f0868474f028fcce8dcfdbeace3 2023-04-28 08:53:56,817 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/83d2fe181f4049e89f07c4e7f11bcd3c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/83d2fe181f4049e89f07c4e7f11bcd3c 2023-04-28 08:53:56,824 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into 83d2fe181f4049e89f07c4e7f11bcd3c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:56,824 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,824 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=9, startTime=1682672036304; duration=0sec 2023-04-28 08:53:56,824 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:56,824 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:56,825 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:56,825 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:56,825 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:56,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:56,826 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,826 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bd16023a68114b49b21d5fdac4e519bd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4cb03e86981c4664b1221e82fd8a68d0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7a80ee07b40d4a829b10818d64018db1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.4 K 2023-04-28 08:53:56,826 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bd16023a68114b49b21d5fdac4e519bd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672033767 2023-04-28 08:53:56,827 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4cb03e86981c4664b1221e82fd8a68d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1682672034915 2023-04-28 08:53:56,827 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7a80ee07b40d4a829b10818d64018db1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682672036063 2023-04-28 08:53:56,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:56,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:56,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:56,829 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:56,829 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,829 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:56,829 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:56,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4cc5993d156f4a73b9ea7829918f797d 2023-04-28 08:53:56,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,832 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,832 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:56,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:56,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/cdf38a1f28414673b582fa6759c83a42 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cdf38a1f28414673b582fa6759c83a42 2023-04-28 08:53:56,838 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#73 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:56,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:56,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:56,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:56,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:56,841 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/0e7b3c9ac152498f841cb378951bfbed as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0e7b3c9ac152498f841cb378951bfbed 2023-04-28 08:53:56,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cdf38a1f28414673b582fa6759c83a42, entries=200, sequenceid=423, filesize=14.3 K 2023-04-28 08:53:56,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:56,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:56,845 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,845 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:56,845 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:56,845 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:56,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:56,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/412d0f0868474f028fcce8dcfdbeace3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/412d0f0868474f028fcce8dcfdbeace3 2023-04-28 08:53:56,846 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:56,846 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:56,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,849 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:56,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2544729a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:56,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:56,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,853 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into 0e7b3c9ac152498f841cb378951bfbed(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:56,853 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,853 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=13, startTime=1682672036304; duration=0sec 2023-04-28 08:53:56,853 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:56,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:56,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:56,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:56,855 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,855 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:56,856 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/412d0f0868474f028fcce8dcfdbeace3, entries=150, sequenceid=423, filesize=11.9 K 2023-04-28 08:53:56,857 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,857 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,857 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,857 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:56,857 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/aa33b6e3f0b140f5aafee415ba966eed as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa33b6e3f0b140f5aafee415ba966eed 2023-04-28 08:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,859 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4cc5993d156f4a73b9ea7829918f797d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4cc5993d156f4a73b9ea7829918f797d 2023-04-28 08:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:56,864 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:56,864 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:56,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:56,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:56,865 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into aa33b6e3f0b140f5aafee415ba966eed(size=12.7 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:56,865 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:56,865 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,865 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=13, startTime=1682672036304; duration=0sec 2023-04-28 08:53:56,865 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:56,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4cc5993d156f4a73b9ea7829918f797d, entries=150, sequenceid=423, filesize=11.9 K 2023-04-28 08:53:56,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dfeafaee81d2f031bddf09740078796 in 557ms, sequenceid=423, compaction requested=false 2023-04-28 08:53:56,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,868 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,870 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,876 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:56,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:56,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:56,877 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:56,877 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:56,877 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,877 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:56,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:56,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:56,957 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 08:53:56,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:56,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:56,959 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:56,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:56,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672096972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672096973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,977 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:56,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672096973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672096974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:56,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:56,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672096977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:56,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:56,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:56,988 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:56,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:56,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:56,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,990 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:56,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:56,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:56,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:56,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:56,992 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,992 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:56,992 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:56,992 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:56,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:56,994 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:56,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:56,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff4b90c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:56,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:56,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,997 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:56,997 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:56,997 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:56,997 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:56,997 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:56,997 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:57,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:57,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:57,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:57,003 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:57,003 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:57,003 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:57,003 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:57,003 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:57,005 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,005 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,005 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:57,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:57,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,013 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:57,013 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:57,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672097079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672097079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672097079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672097079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672097079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,114 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:57,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:57,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:57,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:57,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:57,117 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:57,117 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,117 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:57,117 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:57,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,119 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:57,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:57,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:57,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:57,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:57,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:57,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:57,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:57,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:57,122 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,122 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:57,122 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:57,122 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:57,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:57,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:57,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:57,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,124 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:57,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e64c646[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:57,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:57,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,130 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:57,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:57,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:57,130 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,130 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:57,131 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:57,132 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:57,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:57,134 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:57,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:57,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:57,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:57,135 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:57,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,139 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,143 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,146 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,146 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:57,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:57,147 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:57,148 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:57,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,248 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:57,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:57,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:57,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:57,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:57,254 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:57,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:57,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:57,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,256 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:57,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:57,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:57,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:57,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:57,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:57,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:57,262 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,262 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:57,262 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:57,262 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:57,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:57,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:57,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:57,264 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,264 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:57,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ac3f15[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:57,267 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:57,267 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,268 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:57,268 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:57,268 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:57,268 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,268 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:57,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:57,283 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:57,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:57,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:57,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:57,283 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:57,288 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,288 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,288 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,289 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672097282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672097284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672097285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672097285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672097285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:57,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:57,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,300 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:57,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,300 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:57,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:57,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/1a705e10178b456bba96b176d880fe33 2023-04-28 08:53:57,400 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:57,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:57,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:57,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:57,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:57,411 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:57,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:57,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:57,412 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,412 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,416 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:57,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:57,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:57,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:57,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:57,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:57,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:57,427 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,427 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:57,427 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:57,427 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:57,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:57,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:57,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:57,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/b39585033a56423190e777f089b5b0fc 2023-04-28 08:53:57,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,432 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30fa4bf8[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:57,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:57,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:57,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:57,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:57,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:57,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:57,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:57,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:57,448 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:57,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:57,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:57,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:57,448 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:57,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:57,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:57,457 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/e51c58cd086f4dabbced186e11678420 2023-04-28 08:53:57,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:57,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:57,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:57,458 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:57,458 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:57,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:57,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/1a705e10178b456bba96b176d880fe33 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1a705e10178b456bba96b176d880fe33 2023-04-28 08:53:57,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1a705e10178b456bba96b176d880fe33, entries=150, sequenceid=447, filesize=11.9 K 2023-04-28 08:53:57,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/b39585033a56423190e777f089b5b0fc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/b39585033a56423190e777f089b5b0fc 2023-04-28 08:53:57,479 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/b39585033a56423190e777f089b5b0fc, entries=150, sequenceid=447, filesize=11.9 K 2023-04-28 08:53:57,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/e51c58cd086f4dabbced186e11678420 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e51c58cd086f4dabbced186e11678420 2023-04-28 08:53:57,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e51c58cd086f4dabbced186e11678420, entries=150, sequenceid=447, filesize=11.9 K 2023-04-28 08:53:57,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6dfeafaee81d2f031bddf09740078796 in 533ms, sequenceid=447, compaction requested=true 2023-04-28 08:53:57,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:57,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:57,490 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:57,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:57,490 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:57,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:57,492 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39782 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:57,492 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:57,492 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:57,492 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,492 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:57,492 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,492 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa33b6e3f0b140f5aafee415ba966eed, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cdf38a1f28414673b582fa6759c83a42, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1a705e10178b456bba96b176d880fe33] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=38.8 K 2023-04-28 08:53:57,492 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0e7b3c9ac152498f841cb378951bfbed, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/412d0f0868474f028fcce8dcfdbeace3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/b39585033a56423190e777f089b5b0fc] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.5 K 2023-04-28 08:53:57,493 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting aa33b6e3f0b140f5aafee415ba966eed, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682672036063 2023-04-28 08:53:57,494 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 0e7b3c9ac152498f841cb378951bfbed, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682672036063 2023-04-28 08:53:57,494 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting cdf38a1f28414673b582fa6759c83a42, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1682672036189 2023-04-28 08:53:57,495 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 412d0f0868474f028fcce8dcfdbeace3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1682672036189 2023-04-28 08:53:57,495 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 1a705e10178b456bba96b176d880fe33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682672036334 2023-04-28 08:53:57,495 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b39585033a56423190e777f089b5b0fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682672036334 2023-04-28 08:53:57,504 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#77 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:57,505 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:57,547 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/43a1c0ccd34247f9af55144f1db93d61 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/43a1c0ccd34247f9af55144f1db93d61 2023-04-28 08:53:57,551 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/b92490fb6cf246a9ba147efa9f5ddb3f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b92490fb6cf246a9ba147efa9f5ddb3f 2023-04-28 08:53:57,558 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:57,559 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into b92490fb6cf246a9ba147efa9f5ddb3f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:57,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:57,559 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:57,559 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=13, startTime=1682672037490; duration=0sec 2023-04-28 08:53:57,559 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:57,560 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:57,561 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into 43a1c0ccd34247f9af55144f1db93d61(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:57,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:57,561 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:57,562 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=13, startTime=1682672037490; duration=0sec 2023-04-28 08:53:57,562 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:57,562 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:57,562 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:53:57,562 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,562 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/83d2fe181f4049e89f07c4e7f11bcd3c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4cc5993d156f4a73b9ea7829918f797d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e51c58cd086f4dabbced186e11678420] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.4 K 2023-04-28 08:53:57,563 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 83d2fe181f4049e89f07c4e7f11bcd3c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1682672036063 2023-04-28 08:53:57,563 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4cc5993d156f4a73b9ea7829918f797d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1682672036189 2023-04-28 08:53:57,564 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e51c58cd086f4dabbced186e11678420, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682672036334 2023-04-28 08:53:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:57,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:57,566 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:57,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:57,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:57,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:57,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:57,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:57,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:57,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:57,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:57,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:57,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:57,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:57,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:57,573 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:57,573 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:57,573 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:53:57,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:57,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:57,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:57,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:57,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:57,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:57,580 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#79 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:57,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:57,597 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:57,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672097627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672097651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672097651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672097651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,662 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/65e1710bf37d4209aa62e0ef587ae167 2023-04-28 08:53:57,664 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c33f60da6731416296f2f756bfccfc3b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c33f60da6731416296f2f756bfccfc3b 2023-04-28 08:53:57,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672097659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,676 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into c33f60da6731416296f2f756bfccfc3b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:57,676 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:57,676 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=13, startTime=1682672037490; duration=0sec 2023-04-28 08:53:57,676 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:57,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672097752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672097761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672097762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672097762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672097791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672097959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672097967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672097991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672097991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:57,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:57,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672097993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,133 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/a26c6870da9d49e5bc3663f21fd9255e 2023-04-28 08:53:58,149 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/1fe51778ab5344ad8c519aedb7bcd4a6 2023-04-28 08:53:58,156 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/65e1710bf37d4209aa62e0ef587ae167 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/65e1710bf37d4209aa62e0ef587ae167 2023-04-28 08:53:58,163 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/65e1710bf37d4209aa62e0ef587ae167, entries=150, sequenceid=462, filesize=11.9 K 2023-04-28 08:53:58,165 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/a26c6870da9d49e5bc3663f21fd9255e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a26c6870da9d49e5bc3663f21fd9255e 2023-04-28 08:53:58,173 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a26c6870da9d49e5bc3663f21fd9255e, entries=150, sequenceid=462, filesize=11.9 K 2023-04-28 08:53:58,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/1fe51778ab5344ad8c519aedb7bcd4a6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1fe51778ab5344ad8c519aedb7bcd4a6 2023-04-28 08:53:58,184 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1fe51778ab5344ad8c519aedb7bcd4a6, entries=150, sequenceid=462, filesize=11.9 K 2023-04-28 08:53:58,186 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 6dfeafaee81d2f031bddf09740078796 in 612ms, sequenceid=462, compaction requested=false 2023-04-28 08:53:58,186 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:58,186 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,186 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:58,186 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:58,186 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:58,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:58,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:58,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:58,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0cb832[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:58,193 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:58,193 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,195 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:58,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:58,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:58,196 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,196 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:58,198 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,199 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:58,199 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,199 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:58,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:58,202 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:58,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:58,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:58,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:58,202 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:58,204 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,204 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,213 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:58,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:58,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:58,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,213 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:58,213 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:58,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,265 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-28 08:53:58,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:58,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:58,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:58,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:58,266 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:58,266 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:58,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:58,288 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/ea8141af64f74c95a764c34bb2589225 2023-04-28 08:53:58,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672098285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672098286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672098294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672098296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672098296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,313 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:58,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:58,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:58,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:58,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:58,317 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:58,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:58,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:58,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,320 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,321 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,321 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,321 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:58,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:58,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:58,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:58,331 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:58,331 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:58,332 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,332 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:58,332 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:58,332 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:58,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:58,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,334 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,335 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,335 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:58,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:58,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329217c3[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:58,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:58,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,339 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:58,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:58,339 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:58,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:58,341 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:58,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:58,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:58,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:58,346 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:58,346 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:58,346 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:58,346 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:58,346 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:58,349 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,349 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,349 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,357 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,357 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:58,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,359 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:58,359 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:58,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672098390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672098391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:58,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:58,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:58,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:58,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:58,463 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:58,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:58,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:58,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,468 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:58,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:58,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:58,469 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:58,469 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,469 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:58,469 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:58,469 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:58,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,470 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:58,470 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:58,470 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:58,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:58,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:58,470 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:58,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,477 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6591c213[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:58,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:58,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:58,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:58,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:58,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:58,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:58,488 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:58,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:58,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:58,491 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:58,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:58,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:58,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:58,491 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:58,493 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,493 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,493 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,494 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,496 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:58,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:58,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:58,500 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:58,501 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:58,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672098593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672098593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,601 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:58,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:58,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:58,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:58,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:58,607 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:58,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:58,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:58,608 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,608 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,610 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,611 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,611 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:58,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:58,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:58,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:58,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:58,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:58,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:58,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:58,617 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,617 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:58,617 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:58,617 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:58,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:58,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:58,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:58,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:58,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756f0d2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:58,624 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:58,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,631 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:58,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:58,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:58,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:58,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:58,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:58,637 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:58,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:58,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:58,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:58,637 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:58,640 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,640 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,640 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:58,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:58,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:58,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:58,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:58,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:58,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:58,647 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:58,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:58,648 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:58,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/cd4a0dd6978e4f70b9289ea1071f2bdc 2023-04-28 08:53:58,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/28bf0165c59e49c59e029cf2575f3e79 2023-04-28 08:53:58,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/ea8141af64f74c95a764c34bb2589225 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ea8141af64f74c95a764c34bb2589225 2023-04-28 08:53:58,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ea8141af64f74c95a764c34bb2589225, entries=200, sequenceid=484, filesize=14.3 K 2023-04-28 08:53:58,736 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/cd4a0dd6978e4f70b9289ea1071f2bdc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/cd4a0dd6978e4f70b9289ea1071f2bdc 2023-04-28 08:53:58,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/cd4a0dd6978e4f70b9289ea1071f2bdc, entries=150, sequenceid=484, filesize=11.9 K 2023-04-28 08:53:58,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/28bf0165c59e49c59e029cf2575f3e79 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28bf0165c59e49c59e029cf2575f3e79 2023-04-28 08:53:58,749 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:58,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:58,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:58,756 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:58,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:58,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:58,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28bf0165c59e49c59e029cf2575f3e79, entries=150, sequenceid=484, filesize=11.9 K 2023-04-28 08:53:58,758 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:58,758 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:58,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 6dfeafaee81d2f031bddf09740078796 in 493ms, sequenceid=484, compaction requested=true 2023-04-28 08:53:58,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:58,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:58,759 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,759 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:58,759 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:58,759 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:58,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:58,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:58,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:58,761 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:58,761 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:58,761 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:53:58,761 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:53:58,761 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,761 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,761 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/43a1c0ccd34247f9af55144f1db93d61, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a26c6870da9d49e5bc3663f21fd9255e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/cd4a0dd6978e4f70b9289ea1071f2bdc] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.6 K 2023-04-28 08:53:58,761 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b92490fb6cf246a9ba147efa9f5ddb3f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/65e1710bf37d4209aa62e0ef587ae167, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ea8141af64f74c95a764c34bb2589225] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=38.9 K 2023-04-28 08:53:58,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:58,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:58,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:58,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:58,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:58,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:58,762 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,762 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:58,762 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:53:58,762 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:58,762 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:58,763 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:58,763 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:58,763 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:58,763 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:53:58,766 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 43a1c0ccd34247f9af55144f1db93d61, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682672036334 2023-04-28 08:53:58,766 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b92490fb6cf246a9ba147efa9f5ddb3f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682672036334 2023-04-28 08:53:58,767 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a26c6870da9d49e5bc3663f21fd9255e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682672036969 2023-04-28 08:53:58,767 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 65e1710bf37d4209aa62e0ef587ae167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682672036969 2023-04-28 08:53:58,767 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting cd4a0dd6978e4f70b9289ea1071f2bdc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682672037619 2023-04-28 08:53:58,768 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ea8141af64f74c95a764c34bb2589225, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682672037619 2023-04-28 08:53:58,796 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:58,796 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:58,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:53:58,821 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/f7c051f7a0c44f59b3e72ccb6bc695ca as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f7c051f7a0c44f59b3e72ccb6bc695ca 2023-04-28 08:53:58,827 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/c0c4a3b3cb664792a4690bccfd89b328 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c0c4a3b3cb664792a4690bccfd89b328 2023-04-28 08:53:58,833 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into f7c051f7a0c44f59b3e72ccb6bc695ca(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:58,833 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:58,833 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=13, startTime=1682672038759; duration=0sec 2023-04-28 08:53:58,833 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:53:58,833 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:53:58,837 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:53:58,837 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:53:58,837 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:58,837 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c33f60da6731416296f2f756bfccfc3b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1fe51778ab5344ad8c519aedb7bcd4a6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28bf0165c59e49c59e029cf2575f3e79] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.5 K 2023-04-28 08:53:58,837 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting c33f60da6731416296f2f756bfccfc3b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1682672036334 2023-04-28 08:53:58,838 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into c0c4a3b3cb664792a4690bccfd89b328(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:58,838 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:58,838 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=13, startTime=1682672038759; duration=0sec 2023-04-28 08:53:58,838 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1fe51778ab5344ad8c519aedb7bcd4a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1682672036969 2023-04-28 08:53:58,838 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:58,839 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 28bf0165c59e49c59e029cf2575f3e79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682672037619 2023-04-28 08:53:58,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672038837, max timestamp=1682672038846Num uniques 859; 2023-04-28 08:53:58,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:53:58,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672038837, max timestamp=1682672038846Num uniques 859; 2023-04-28 08:53:58,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:53:58,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672038837, max timestamp=1682672038846Num uniques 859; 2023-04-28 08:53:58,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:53:58,859 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:53:58,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672098871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672098872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672098875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,876 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/04b3e78df2da46e2a5048568b28fa6d0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/04b3e78df2da46e2a5048568b28fa6d0 2023-04-28 08:53:58,882 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into 04b3e78df2da46e2a5048568b28fa6d0(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:53:58,882 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:58,882 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=13, startTime=1682672038759; duration=0sec 2023-04-28 08:53:58,882 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:53:58,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672098896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672098896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672098976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672098976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:58,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:58,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672098976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672099179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672099180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672099181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,214 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/bb8c363952224776924bb291f7e46f74 2023-04-28 08:53:59,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672099402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672099402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672099483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672099483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:53:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672099484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,654 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/527e648647d84f31a78331184de6c2fa 2023-04-28 08:53:59,682 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=495 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/43a9f50f78804286a6244304ef5632c2 2023-04-28 08:53:59,694 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/bb8c363952224776924bb291f7e46f74 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bb8c363952224776924bb291f7e46f74 2023-04-28 08:53:59,704 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bb8c363952224776924bb291f7e46f74, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 08:53:59,705 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/527e648647d84f31a78331184de6c2fa as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/527e648647d84f31a78331184de6c2fa 2023-04-28 08:53:59,715 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/527e648647d84f31a78331184de6c2fa, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 08:53:59,716 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/43a9f50f78804286a6244304ef5632c2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/43a9f50f78804286a6244304ef5632c2 2023-04-28 08:53:59,721 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/43a9f50f78804286a6244304ef5632c2, entries=150, sequenceid=495, filesize=11.9 K 2023-04-28 08:53:59,722 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 6dfeafaee81d2f031bddf09740078796 in 960ms, sequenceid=495, compaction requested=false 2023-04-28 08:53:59,722 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:53:59,722 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:59,722 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:53:59,722 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:53:59,722 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,722 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:53:59,722 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:53:59,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:59,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:59,725 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,725 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:53:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:59,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:59,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:53:59,728 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:53:59,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18840522[Count = 0] remaining members to acquire global barrier 2023-04-28 08:53:59,728 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,729 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:53:59,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:53:59,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:53:59,730 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,730 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:53:59,733 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:53:59,733 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:59,733 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:53:59,733 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:53:59,743 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:53:59,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:53:59,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:53:59,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:53:59,743 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:53:59,745 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,745 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,746 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:53:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,746 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:53:59,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,753 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:59,753 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:53:59,753 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:59,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:53:59,753 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:59,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:59,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:53:59,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:53:59,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:53:59,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:53:59,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:53:59,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:53:59,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:53:59,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:53:59,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:53:59,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:53:59,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:53:59,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:53:59,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:53:59,859 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:53:59,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:59,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:53:59,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:53:59,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:59,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:53:59,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:53:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:53:59,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:59,862 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:53:59,862 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:53:59,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:53:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:53:59,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:53:59,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:53:59,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:53:59,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:53:59,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:53:59,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:53:59,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:53:59,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:53:59,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:53:59,863 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:53:59,863 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:53:59,863 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-28 08:53:59,864 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:53:59,864 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:59,864 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:53:59,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:59,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:53:59,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:53:59,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:53:59,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:00,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672100012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672100015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672100015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672100117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672100118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672100118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,282 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/737cada8ecb6489d9f97c9213eb195a2 2023-04-28 08:54:00,297 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/54d054297e5c48df9f1ac2b2c9bd7295 2023-04-28 08:54:00,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672100319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672100320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672100320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,328 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/0f962fe8b23349279ef379cea87557cb 2023-04-28 08:54:00,340 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/737cada8ecb6489d9f97c9213eb195a2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/737cada8ecb6489d9f97c9213eb195a2 2023-04-28 08:54:00,349 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/737cada8ecb6489d9f97c9213eb195a2, entries=150, sequenceid=532, filesize=11.9 K 2023-04-28 08:54:00,350 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/54d054297e5c48df9f1ac2b2c9bd7295 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/54d054297e5c48df9f1ac2b2c9bd7295 2023-04-28 08:54:00,355 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/54d054297e5c48df9f1ac2b2c9bd7295, entries=150, sequenceid=532, filesize=11.9 K 2023-04-28 08:54:00,356 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/0f962fe8b23349279ef379cea87557cb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0f962fe8b23349279ef379cea87557cb 2023-04-28 08:54:00,372 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0f962fe8b23349279ef379cea87557cb, entries=150, sequenceid=532, filesize=11.9 K 2023-04-28 08:54:00,373 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 6dfeafaee81d2f031bddf09740078796 in 510ms, sequenceid=532, compaction requested=true 2023-04-28 08:54:00,374 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:00,374 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:00,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:00,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:00,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:00,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:00,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a8a35fe[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:00,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:00,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:00,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:00,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:00,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:00,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:00,384 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:00,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:00,387 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:00,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:00,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:00,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:00,387 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:00,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,392 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:00,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:00,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,395 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,395 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:00,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:00,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:54:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:54:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:54:00,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:00,412 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:54:00,412 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:00,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/066dcde9580840ee8f05a63c3891cddc 2023-04-28 08:54:00,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672100445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672100445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,496 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:00,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:00,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:00,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:00,503 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:00,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:00,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:00,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,505 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,505 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:00,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,506 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:00,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:00,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:00,507 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:00,507 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,507 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:00,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:00,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:00,508 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,508 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:00,508 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:00,508 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:00,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:00,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:00,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:00,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:00,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5160f9e6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:00,514 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:00,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,516 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,516 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:00,516 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:00,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:00,517 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,517 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:00,519 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,519 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,519 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,519 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:00,524 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:00,524 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:00,524 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:00,524 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:00,524 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:00,527 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,527 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,527 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,531 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:00,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:00,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,535 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:00,535 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:00,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672100550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672100550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672100623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672100623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672100626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,635 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:00,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:00,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:00,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:00,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:00,639 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:00,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:00,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:00,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:00,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:00,643 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,643 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:00,643 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:00,643 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:00,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:00,645 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61805378[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:00,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:00,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:00,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:00,648 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:00,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:00,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:00,650 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:00,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:00,651 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:00,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:00,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:00,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:00,651 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:00,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,656 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,660 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,660 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,660 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,660 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:00,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:00,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:00,660 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,661 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:00,661 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:00,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672100754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672100754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,761 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:00,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:00,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:00,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:00,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:00,764 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:00,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:00,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:00,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,766 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:00,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:00,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:00,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:00,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:00,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:00,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:00,770 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,770 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:00,770 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:00,770 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:00,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:00,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:00,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:00,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,774 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5995f77f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:00,775 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:00,775 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:00,777 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:00,777 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:00,777 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:00,781 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:00,784 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:00,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:00,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:00,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:00,784 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:00,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,787 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,787 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:00,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:00,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:00,791 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:00,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,791 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:00,791 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:00,791 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:00,792 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/dc3535088096487e9a9f9ca9975720ae 2023-04-28 08:54:00,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4c042dae1a6e463eac952fee38e904f1 2023-04-28 08:54:00,857 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/066dcde9580840ee8f05a63c3891cddc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/066dcde9580840ee8f05a63c3891cddc 2023-04-28 08:54:00,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/066dcde9580840ee8f05a63c3891cddc, entries=200, sequenceid=545, filesize=14.3 K 2023-04-28 08:54:00,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/dc3535088096487e9a9f9ca9975720ae as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/dc3535088096487e9a9f9ca9975720ae 2023-04-28 08:54:00,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/dc3535088096487e9a9f9ca9975720ae, entries=150, sequenceid=545, filesize=11.9 K 2023-04-28 08:54:00,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4c042dae1a6e463eac952fee38e904f1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4c042dae1a6e463eac952fee38e904f1 2023-04-28 08:54:00,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4c042dae1a6e463eac952fee38e904f1, entries=150, sequenceid=545, filesize=11.9 K 2023-04-28 08:54:00,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6dfeafaee81d2f031bddf09740078796 in 464ms, sequenceid=545, compaction requested=true 2023-04-28 08:54:00,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:00,875 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:00,875 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:00,875 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:00,875 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:00,875 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:00,878 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52172 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:00,878 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:54:00,878 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,878 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f7c051f7a0c44f59b3e72ccb6bc695ca, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bb8c363952224776924bb291f7e46f74, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/737cada8ecb6489d9f97c9213eb195a2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/066dcde9580840ee8f05a63c3891cddc] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=50.9 K 2023-04-28 08:54:00,879 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f7c051f7a0c44f59b3e72ccb6bc695ca, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682672037619 2023-04-28 08:54:00,879 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:00,879 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:54:00,879 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,879 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c0c4a3b3cb664792a4690bccfd89b328, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/527e648647d84f31a78331184de6c2fa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/54d054297e5c48df9f1ac2b2c9bd7295, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/dc3535088096487e9a9f9ca9975720ae] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=48.6 K 2023-04-28 08:54:00,879 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bb8c363952224776924bb291f7e46f74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682672038276 2023-04-28 08:54:00,880 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c0c4a3b3cb664792a4690bccfd89b328, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682672037619 2023-04-28 08:54:00,880 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 737cada8ecb6489d9f97c9213eb195a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682672038870 2023-04-28 08:54:00,880 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 527e648647d84f31a78331184de6c2fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682672038276 2023-04-28 08:54:00,880 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 066dcde9580840ee8f05a63c3891cddc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682672040001 2023-04-28 08:54:00,880 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 54d054297e5c48df9f1ac2b2c9bd7295, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682672038870 2023-04-28 08:54:00,881 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dc3535088096487e9a9f9ca9975720ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682672040006 2023-04-28 08:54:00,891 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:00,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:00,893 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:00,893 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:00,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:00,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:00,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:00,895 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:00,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:00,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:00,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:00,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:00,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:00,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:00,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:00,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:00,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:00,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:00,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:00,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:00,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:00,899 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,899 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:00,901 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 08:54:00,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:54:00,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:00,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:54:00,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:00,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:54:00,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:00,905 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/be0f3dd7f05f4a05977542db33d5541f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/be0f3dd7f05f4a05977542db33d5541f 2023-04-28 08:54:00,914 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into be0f3dd7f05f4a05977542db33d5541f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:00,914 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:00,914 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=12, startTime=1682672040875; duration=0sec 2023-04-28 08:54:00,914 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:00,914 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:00,914 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/f30839cd7df748c1a12abd9f265deed4 2023-04-28 08:54:00,916 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:00,917 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:54:00,917 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:00,917 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/04b3e78df2da46e2a5048568b28fa6d0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/43a9f50f78804286a6244304ef5632c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0f962fe8b23349279ef379cea87557cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4c042dae1a6e463eac952fee38e904f1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=48.5 K 2023-04-28 08:54:00,917 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 04b3e78df2da46e2a5048568b28fa6d0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1682672037619 2023-04-28 08:54:00,919 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 43a9f50f78804286a6244304ef5632c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=495, earliestPutTs=1682672038276 2023-04-28 08:54:00,919 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0f962fe8b23349279ef379cea87557cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1682672038870 2023-04-28 08:54:00,920 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4c042dae1a6e463eac952fee38e904f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682672040006 2023-04-28 08:54:00,921 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/cd1f87e73d754d1eae393d996f95f954 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cd1f87e73d754d1eae393d996f95f954 2023-04-28 08:54:00,927 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into cd1f87e73d754d1eae393d996f95f954(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:00,928 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:00,928 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=12, startTime=1682672040875; duration=0sec 2023-04-28 08:54:00,928 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:00,931 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#102 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:00,932 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/24a4a0ace2ba4202a829758d9060cb4b 2023-04-28 08:54:00,952 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/d256c9fe65f84120b241578def624e77 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d256c9fe65f84120b241578def624e77 2023-04-28 08:54:00,960 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into d256c9fe65f84120b241578def624e77(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:00,960 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:00,960 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=12, startTime=1682672040875; duration=0sec 2023-04-28 08:54:00,960 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:01,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:01,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:01,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672101069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672101071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672101130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672101130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672101132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672101173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672101174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,351 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4e96511b5c1c4960ae987de08c1b4fcf 2023-04-28 08:54:01,357 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/f30839cd7df748c1a12abd9f265deed4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f30839cd7df748c1a12abd9f265deed4 2023-04-28 08:54:01,362 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f30839cd7df748c1a12abd9f265deed4, entries=150, sequenceid=564, filesize=11.9 K 2023-04-28 08:54:01,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/24a4a0ace2ba4202a829758d9060cb4b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/24a4a0ace2ba4202a829758d9060cb4b 2023-04-28 08:54:01,369 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/24a4a0ace2ba4202a829758d9060cb4b, entries=150, sequenceid=564, filesize=11.9 K 2023-04-28 08:54:01,370 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4e96511b5c1c4960ae987de08c1b4fcf as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4e96511b5c1c4960ae987de08c1b4fcf 2023-04-28 08:54:01,375 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4e96511b5c1c4960ae987de08c1b4fcf, entries=150, sequenceid=564, filesize=11.9 K 2023-04-28 08:54:01,376 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6dfeafaee81d2f031bddf09740078796 in 477ms, sequenceid=564, compaction requested=false 2023-04-28 08:54:01,376 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:01,377 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:01,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:01,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:01,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:01,379 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:01,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:01,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 08:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:54:01,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:01,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:01,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:54:01,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325b1c76[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:01,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:01,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,384 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:01,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:01,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:01,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:01,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:01,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:01,387 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:01,390 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:01,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:01,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:01,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:01,390 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:01,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,393 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,399 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:01,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:01,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:01,398 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:01,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/823900a72c794ceb88e55a0b82972af1 2023-04-28 08:54:01,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672101412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,415 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672101413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,499 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:01,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:01,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:01,503 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:01,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:01,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:01,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:01,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:01,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:01,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:01,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:01,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:01,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:01,512 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,512 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:01,513 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:01,513 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,513 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:01,513 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:01,513 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,513 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:01,513 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:01,515 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,515 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,515 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,515 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:01,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672101516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672101516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:01,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64226c12[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:01,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:01,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,540 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:01,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:01,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:01,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:01,542 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:01,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,542 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,542 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:01,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:01,544 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:01,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:01,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:01,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:01,544 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:01,547 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,547 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,547 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,548 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,552 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:01,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:01,552 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:01,552 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:01,554 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:01,653 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:01,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:01,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:01,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:01,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:01,657 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:01,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:01,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:01,658 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,658 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,659 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,659 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:01,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:01,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:01,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:01,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:01,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:01,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:01,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:01,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:01,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:01,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:01,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:01,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:01,663 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,664 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,664 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:01,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dae87d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:01,665 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:01,665 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,667 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,668 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:01,668 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:01,668 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:01,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:01,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,671 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,671 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,671 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:01,675 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:01,675 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:01,675 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:01,675 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:01,675 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:01,677 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,677 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,689 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:01,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,690 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:01,690 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:01,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:01,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672101719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:01,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672101738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,790 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:01,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:01,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:01,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:01,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:01,794 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:01,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:01,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:01,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,796 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:01,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:01,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:01,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:01,798 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,798 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:01,798 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:01,798 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:01,798 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:01,800 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,801 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,801 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:01,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7102d9f6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:01,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:01,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,804 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:01,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:01,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:01,806 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:01,806 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,806 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:01,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:01,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:01,807 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:01,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:01,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:01,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:01,807 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:01,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,827 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:01,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:01,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:01,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,831 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:01,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,831 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,831 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:01,835 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/55b2d5f08b38479b8f89e7f48e04aba9 2023-04-28 08:54:01,932 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:01,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:01,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:01,935 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:01,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:01,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:01,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,937 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,937 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,937 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:01,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:01,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:01,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:01,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:01,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:01,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:01,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:01,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:01,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:01,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:01,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:01,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:01,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:01,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,941 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,942 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,942 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:01,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bd0c30c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:01,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:01,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:01,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:01,946 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,946 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:01,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:01,947 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:01,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:01,950 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:01,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:01,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:01,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:01,950 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:01,952 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,952 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,952 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:01,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,953 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:01,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:01,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:01,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:01,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:01,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:01,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:01,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:01,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:01,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:01,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:01,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:01,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:01,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:01,961 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:01,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:02,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672102023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672102041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:02,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:02,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:02,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:02,064 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:02,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:02,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:02,065 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:02,065 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,065 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:02,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:02,067 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,067 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:02,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:02,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:02,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:02,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:02,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:02,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:02,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:02,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:02,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:02,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:02,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:02,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:02,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:02,077 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:02,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:02,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:02,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:02,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:02,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce57e98[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:02,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:02,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:02,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:02,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:02,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:02,083 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:02,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:02,085 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:02,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:02,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:02,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:02,086 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:02,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:02,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:02,088 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,089 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:02,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:02,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:02,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:02,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:02,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:02,095 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:02,095 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:02,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:02,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:02,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59032 deadline: 1682672102132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59024 deadline: 1682672102139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59076 deadline: 1682672102142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,195 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:02,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:02,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:02,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:02,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:02,198 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:02,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:02,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:02,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:02,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,203 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,203 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:02,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:02,204 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,204 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:02,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:02,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:02,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:02,205 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,205 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:02,205 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:02,205 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:02,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:02,207 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:02,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:02,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:02,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c2cea79[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:02,209 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:02,209 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:02,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,212 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:02,212 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:02,212 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:02,213 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:02,213 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:02,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,213 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:02,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:02,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:02,221 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:02,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:02,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:02,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:02,221 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:02,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:02,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:02,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:02,225 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,227 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:02,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:02,232 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:02,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:02,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:02,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:02,233 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,233 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:02,233 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:02,233 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:02,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:02,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4f4e2f53bb0c4aac8915fd2917543490 2023-04-28 08:54:02,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/823900a72c794ceb88e55a0b82972af1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/823900a72c794ceb88e55a0b82972af1 2023-04-28 08:54:02,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/823900a72c794ceb88e55a0b82972af1, entries=200, sequenceid=582, filesize=14.3 K 2023-04-28 08:54:02,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/55b2d5f08b38479b8f89e7f48e04aba9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/55b2d5f08b38479b8f89e7f48e04aba9 2023-04-28 08:54:02,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/55b2d5f08b38479b8f89e7f48e04aba9, entries=150, sequenceid=582, filesize=11.9 K 2023-04-28 08:54:02,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/4f4e2f53bb0c4aac8915fd2917543490 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4f4e2f53bb0c4aac8915fd2917543490 2023-04-28 08:54:02,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4f4e2f53bb0c4aac8915fd2917543490, entries=150, sequenceid=582, filesize=11.9 K 2023-04-28 08:54:02,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6dfeafaee81d2f031bddf09740078796 in 907ms, sequenceid=582, compaction requested=true 2023-04-28 08:54:02,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:02,286 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:02,287 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:02,287 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:02,287 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:02,287 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:02,288 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:02,288 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:02,288 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/B is initiating minor compaction (all files) 2023-04-28 08:54:02,288 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/A is initiating minor compaction (all files) 2023-04-28 08:54:02,289 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/B in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,289 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/A in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,289 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/be0f3dd7f05f4a05977542db33d5541f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/24a4a0ace2ba4202a829758d9060cb4b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/55b2d5f08b38479b8f89e7f48e04aba9] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.8 K 2023-04-28 08:54:02,289 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cd1f87e73d754d1eae393d996f95f954, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f30839cd7df748c1a12abd9f265deed4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/823900a72c794ceb88e55a0b82972af1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=39.2 K 2023-04-28 08:54:02,289 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting be0f3dd7f05f4a05977542db33d5541f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682672040006 2023-04-28 08:54:02,289 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting cd1f87e73d754d1eae393d996f95f954, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682672040006 2023-04-28 08:54:02,290 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 24a4a0ace2ba4202a829758d9060cb4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1682672040440 2023-04-28 08:54:02,290 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f30839cd7df748c1a12abd9f265deed4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1682672040440 2023-04-28 08:54:02,290 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 55b2d5f08b38479b8f89e7f48e04aba9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672041065 2023-04-28 08:54:02,291 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 823900a72c794ceb88e55a0b82972af1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672041065 2023-04-28 08:54:02,309 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#A#compaction#107 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:02,309 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#B#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:02,333 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:02,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:02,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:02,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:02,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:02,339 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:02,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:02,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:02,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:02,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,340 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,341 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:02,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:02,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:02,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:02,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:02,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:02,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:02,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:02,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:02,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:02,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:02,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. started... 2023-04-28 08:54:02,343 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 08:54:02,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:54:02,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:02,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:54:02,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:02,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:54:02,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:02,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:02,529 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. as already flushing 2023-04-28 08:54:02,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672102550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672102552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,556 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:54655 2023-04-28 08:54:02,556 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:02,557 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:54655 2023-04-28 08:54:02,557 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:02,559 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:54655 2023-04-28 08:54:02,559 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:02,563 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:54655 2023-04-28 08:54:02,563 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:02,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672102654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672102656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,729 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/9afc635074f24b83b99ec062f71ceefe as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9afc635074f24b83b99ec062f71ceefe 2023-04-28 08:54:02,730 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/4b8f3f7e098a4f9e9566992b066a6378 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4b8f3f7e098a4f9e9566992b066a6378 2023-04-28 08:54:02,738 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/B of 6dfeafaee81d2f031bddf09740078796 into 9afc635074f24b83b99ec062f71ceefe(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:02,738 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:02,738 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/B, priority=13, startTime=1682672042287; duration=0sec 2023-04-28 08:54:02,739 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:02,739 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:02,739 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/A of 6dfeafaee81d2f031bddf09740078796 into 4b8f3f7e098a4f9e9566992b066a6378(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:02,739 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:02,739 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/A, priority=13, startTime=1682672042286; duration=0sec 2023-04-28 08:54:02,740 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:02,740 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:02,740 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 6dfeafaee81d2f031bddf09740078796/C is initiating minor compaction (all files) 2023-04-28 08:54:02,740 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dfeafaee81d2f031bddf09740078796/C in TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:02,740 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d256c9fe65f84120b241578def624e77, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4e96511b5c1c4960ae987de08c1b4fcf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4f4e2f53bb0c4aac8915fd2917543490] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp, totalSize=36.8 K 2023-04-28 08:54:02,740 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d256c9fe65f84120b241578def624e77, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1682672040006 2023-04-28 08:54:02,741 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4e96511b5c1c4960ae987de08c1b4fcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1682672040440 2023-04-28 08:54:02,741 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4f4e2f53bb0c4aac8915fd2917543490, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672041065 2023-04-28 08:54:02,748 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dfeafaee81d2f031bddf09740078796#C#compaction#110 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:02,758 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/0c473ba1e0bd4516944a42600903cd09 2023-04-28 08:54:02,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672102855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:02,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:02,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672102857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:03,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59062 deadline: 1682672103159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:03,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59018 deadline: 1682672103162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,171 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/14a5f2ecd6414b9ba8a58d81f660a761 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/14a5f2ecd6414b9ba8a58d81f660a761 2023-04-28 08:54:03,173 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/883b57cc4d3c4ff4abf5e07afe51ae81 2023-04-28 08:54:03,177 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dfeafaee81d2f031bddf09740078796/C of 6dfeafaee81d2f031bddf09740078796 into 14a5f2ecd6414b9ba8a58d81f660a761(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:03,177 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:03,177 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796., storeName=6dfeafaee81d2f031bddf09740078796/C, priority=13, startTime=1682672042287; duration=0sec 2023-04-28 08:54:03,178 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:03,604 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=599 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c8907768855b4402bb867677e861a7ca 2023-04-28 08:54:03,608 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/0c473ba1e0bd4516944a42600903cd09 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/0c473ba1e0bd4516944a42600903cd09 2023-04-28 08:54:03,613 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/0c473ba1e0bd4516944a42600903cd09, entries=150, sequenceid=599, filesize=11.9 K 2023-04-28 08:54:03,614 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/883b57cc4d3c4ff4abf5e07afe51ae81 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/883b57cc4d3c4ff4abf5e07afe51ae81 2023-04-28 08:54:03,619 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/883b57cc4d3c4ff4abf5e07afe51ae81, entries=150, sequenceid=599, filesize=11.9 K 2023-04-28 08:54:03,620 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c8907768855b4402bb867677e861a7ca as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c8907768855b4402bb867677e861a7ca 2023-04-28 08:54:03,625 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c8907768855b4402bb867677e861a7ca, entries=150, sequenceid=599, filesize=11.9 K 2023-04-28 08:54:03,626 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 6dfeafaee81d2f031bddf09740078796 in 1283ms, sequenceid=599, compaction requested=false 2023-04-28 08:54:03,626 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:03,626 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:03,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:03,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:03,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:03,626 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:03,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,628 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:03,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:03,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:03,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:03,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b864963[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:03,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:03,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,631 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:03,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:03,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:03,632 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,632 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:03,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:03,633 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:03,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:03,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:03,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:03,636 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:03,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:03,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:03,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:03,636 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:03,637 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:03,637 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:03,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:03,637 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:03,638 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:03,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:03,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:03,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:03,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:03,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:03,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:03,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:03,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:03,642 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:03,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:03,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:03,642 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:03,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:54:03,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:54:03,663 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:54655 2023-04-28 08:54:03,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:03,663 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:03,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:54:03,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:03,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:54:03,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:03,665 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:54655 2023-04-28 08:54:03,665 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:03,743 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:03,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:04,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/7600b6424485494683fe900fd68e834c 2023-04-28 08:54:04,144 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:54655 2023-04-28 08:54:04,144 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:04,152 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:54655 2023-04-28 08:54:04,153 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:04,155 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:54655 2023-04-28 08:54:04,155 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:04,155 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 08:54:04,155 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 95 2023-04-28 08:54:04,155 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 91 2023-04-28 08:54:04,155 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-28 08:54:04,155 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 113 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 3884 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 3811 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1695 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5078 rows 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1687 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5060 rows 2023-04-28 08:54:04,156 INFO [Listener at localhost/39717] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 08:54:04,156 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:54655 2023-04-28 08:54:04,156 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:04,160 INFO [Listener at localhost/39717] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 08:54:04,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable TestAcidGuarantees 2023-04-28 08:54:04,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:04,175 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672044175"}]},"ts":"1682672044175"} 2023-04-28 08:54:04,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 08:54:04,177 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 08:54:04,179 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 08:54:04,180 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dfeafaee81d2f031bddf09740078796, UNASSIGN}] 2023-04-28 08:54:04,181 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dfeafaee81d2f031bddf09740078796, UNASSIGN 2023-04-28 08:54:04,182 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6dfeafaee81d2f031bddf09740078796, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:04,182 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672044182"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672044182"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672044182"}]},"ts":"1682672044182"} 2023-04-28 08:54:04,184 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:54:04,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 08:54:04,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:04,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 6dfeafaee81d2f031bddf09740078796, disabling compactions & flushes 2023-04-28 08:54:04,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:04,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 08:54:04,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/e51be93a91474d618f5ef51e83e2aa8b 2023-04-28 08:54:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 08:54:04,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/d78c0ec86c8a4c6abe0f4de1250c98c8 2023-04-28 08:54:04,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/7600b6424485494683fe900fd68e834c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7600b6424485494683fe900fd68e834c 2023-04-28 08:54:04,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7600b6424485494683fe900fd68e834c, entries=150, sequenceid=618, filesize=11.9 K 2023-04-28 08:54:04,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/e51be93a91474d618f5ef51e83e2aa8b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e51be93a91474d618f5ef51e83e2aa8b 2023-04-28 08:54:04,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e51be93a91474d618f5ef51e83e2aa8b, entries=150, sequenceid=618, filesize=11.9 K 2023-04-28 08:54:04,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/d78c0ec86c8a4c6abe0f4de1250c98c8 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d78c0ec86c8a4c6abe0f4de1250c98c8 2023-04-28 08:54:04,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d78c0ec86c8a4c6abe0f4de1250c98c8, entries=150, sequenceid=618, filesize=11.9 K 2023-04-28 08:54:04,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=26.84 KB/27480 for 6dfeafaee81d2f031bddf09740078796 in 1257ms, sequenceid=618, compaction requested=true 2023-04-28 08:54:04,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:04,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:04,919 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. because compaction request was cancelled 2023-04-28 08:54:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:04,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. after waiting 0 ms 2023-04-28 08:54:04,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:04,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:04,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:04,920 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. because compaction request was cancelled 2023-04-28 08:54:04,920 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 6dfeafaee81d2f031bddf09740078796 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 08:54:04,919 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. because compaction request was cancelled 2023-04-28 08:54:04,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=A 2023-04-28 08:54:04,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:04,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=B 2023-04-28 08:54:04,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:04,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dfeafaee81d2f031bddf09740078796, store=C 2023-04-28 08:54:04,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:05,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 08:54:05,327 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/ee2a087bbd3e4f4c846cf8fac2719966 2023-04-28 08:54:05,739 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/9e41716579d84771afd052c973262dc2 2023-04-28 08:54:06,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c6f1df8ee0404bf1939ff4595c79f993 2023-04-28 08:54:06,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/A/ee2a087bbd3e4f4c846cf8fac2719966 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ee2a087bbd3e4f4c846cf8fac2719966 2023-04-28 08:54:06,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ee2a087bbd3e4f4c846cf8fac2719966, entries=100, sequenceid=625, filesize=9.5 K 2023-04-28 08:54:06,159 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/B/9e41716579d84771afd052c973262dc2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9e41716579d84771afd052c973262dc2 2023-04-28 08:54:06,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9e41716579d84771afd052c973262dc2, entries=100, sequenceid=625, filesize=9.5 K 2023-04-28 08:54:06,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/.tmp/C/c6f1df8ee0404bf1939ff4595c79f993 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c6f1df8ee0404bf1939ff4595c79f993 2023-04-28 08:54:06,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c6f1df8ee0404bf1939ff4595c79f993, entries=100, sequenceid=625, filesize=9.5 K 2023-04-28 08:54:06,168 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 6dfeafaee81d2f031bddf09740078796 in 1248ms, sequenceid=625, compaction requested=true 2023-04-28 08:54:06,189 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b5334359a7da4758a0fae409644e23e0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa7c555d7fe547aa9e7475240280e376, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/a4caf73cba314f518eff75bee39d81c5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/6cfdb64ebe404325a2ac9f0cec577ce2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/17756b75d65840a794a7a45451fabf4a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4ee24e314e2e438390ff91df14a92bf2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f88e2c5e0f014fb2b1d6c8b75c76e1de, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/81422f1c28ff48088bb88343cabf36db, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/756a5ce882b248adaae91ca51c0b1a54, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/9e3471d473e94ad4beb75f0a5ff045f2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/04aa814dcfe040079ee16e5ed3f50867, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7aed17c7eb3a4cda824a5652d4436199, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/56c49e996f2d4571aa5a11af61022ea1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1b5487446c5246b6b2d5f74c1d864d28, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/fa0ad527748b41a489dd3b423cc3da48, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aed6111d1b684bfeb86e0ba39ff45af4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/50f16544715842d38f9c37ef4e8811a7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/26fa5325d1b54e01ba9bba41e6d6448a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bd16023a68114b49b21d5fdac4e519bd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/63a38b82ffba41f9b2aaa2196b7a1e79, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4cb03e86981c4664b1221e82fd8a68d0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa33b6e3f0b140f5aafee415ba966eed, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7a80ee07b40d4a829b10818d64018db1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cdf38a1f28414673b582fa6759c83a42, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b92490fb6cf246a9ba147efa9f5ddb3f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1a705e10178b456bba96b176d880fe33, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/65e1710bf37d4209aa62e0ef587ae167, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ea8141af64f74c95a764c34bb2589225, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f7c051f7a0c44f59b3e72ccb6bc695ca, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bb8c363952224776924bb291f7e46f74, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/737cada8ecb6489d9f97c9213eb195a2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/066dcde9580840ee8f05a63c3891cddc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cd1f87e73d754d1eae393d996f95f954, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f30839cd7df748c1a12abd9f265deed4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/823900a72c794ceb88e55a0b82972af1] to archive 2023-04-28 08:54:06,190 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:54:06,196 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b5334359a7da4758a0fae409644e23e0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b5334359a7da4758a0fae409644e23e0 2023-04-28 08:54:06,198 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa7c555d7fe547aa9e7475240280e376 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa7c555d7fe547aa9e7475240280e376 2023-04-28 08:54:06,199 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/a4caf73cba314f518eff75bee39d81c5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/a4caf73cba314f518eff75bee39d81c5 2023-04-28 08:54:06,200 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/6cfdb64ebe404325a2ac9f0cec577ce2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/6cfdb64ebe404325a2ac9f0cec577ce2 2023-04-28 08:54:06,201 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/17756b75d65840a794a7a45451fabf4a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/17756b75d65840a794a7a45451fabf4a 2023-04-28 08:54:06,203 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4ee24e314e2e438390ff91df14a92bf2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4ee24e314e2e438390ff91df14a92bf2 2023-04-28 08:54:06,204 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f88e2c5e0f014fb2b1d6c8b75c76e1de to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f88e2c5e0f014fb2b1d6c8b75c76e1de 2023-04-28 08:54:06,205 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/81422f1c28ff48088bb88343cabf36db to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/81422f1c28ff48088bb88343cabf36db 2023-04-28 08:54:06,206 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/756a5ce882b248adaae91ca51c0b1a54 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/756a5ce882b248adaae91ca51c0b1a54 2023-04-28 08:54:06,207 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/9e3471d473e94ad4beb75f0a5ff045f2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/9e3471d473e94ad4beb75f0a5ff045f2 2023-04-28 08:54:06,208 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/04aa814dcfe040079ee16e5ed3f50867 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/04aa814dcfe040079ee16e5ed3f50867 2023-04-28 08:54:06,209 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7aed17c7eb3a4cda824a5652d4436199 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7aed17c7eb3a4cda824a5652d4436199 2023-04-28 08:54:06,210 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/56c49e996f2d4571aa5a11af61022ea1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/56c49e996f2d4571aa5a11af61022ea1 2023-04-28 08:54:06,212 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1b5487446c5246b6b2d5f74c1d864d28 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1b5487446c5246b6b2d5f74c1d864d28 2023-04-28 08:54:06,213 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/fa0ad527748b41a489dd3b423cc3da48 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/fa0ad527748b41a489dd3b423cc3da48 2023-04-28 08:54:06,215 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aed6111d1b684bfeb86e0ba39ff45af4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aed6111d1b684bfeb86e0ba39ff45af4 2023-04-28 08:54:06,216 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/5b4d1f5eb02d4dc4a88ee20aaeaec40d 2023-04-28 08:54:06,217 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/50f16544715842d38f9c37ef4e8811a7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/50f16544715842d38f9c37ef4e8811a7 2023-04-28 08:54:06,219 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/26fa5325d1b54e01ba9bba41e6d6448a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/26fa5325d1b54e01ba9bba41e6d6448a 2023-04-28 08:54:06,220 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bd16023a68114b49b21d5fdac4e519bd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bd16023a68114b49b21d5fdac4e519bd 2023-04-28 08:54:06,221 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/63a38b82ffba41f9b2aaa2196b7a1e79 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/63a38b82ffba41f9b2aaa2196b7a1e79 2023-04-28 08:54:06,222 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4cb03e86981c4664b1221e82fd8a68d0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4cb03e86981c4664b1221e82fd8a68d0 2023-04-28 08:54:06,226 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa33b6e3f0b140f5aafee415ba966eed to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/aa33b6e3f0b140f5aafee415ba966eed 2023-04-28 08:54:06,228 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7a80ee07b40d4a829b10818d64018db1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7a80ee07b40d4a829b10818d64018db1 2023-04-28 08:54:06,229 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cdf38a1f28414673b582fa6759c83a42 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cdf38a1f28414673b582fa6759c83a42 2023-04-28 08:54:06,230 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b92490fb6cf246a9ba147efa9f5ddb3f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/b92490fb6cf246a9ba147efa9f5ddb3f 2023-04-28 08:54:06,231 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1a705e10178b456bba96b176d880fe33 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/1a705e10178b456bba96b176d880fe33 2023-04-28 08:54:06,232 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/65e1710bf37d4209aa62e0ef587ae167 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/65e1710bf37d4209aa62e0ef587ae167 2023-04-28 08:54:06,234 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ea8141af64f74c95a764c34bb2589225 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ea8141af64f74c95a764c34bb2589225 2023-04-28 08:54:06,235 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f7c051f7a0c44f59b3e72ccb6bc695ca to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f7c051f7a0c44f59b3e72ccb6bc695ca 2023-04-28 08:54:06,236 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bb8c363952224776924bb291f7e46f74 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/bb8c363952224776924bb291f7e46f74 2023-04-28 08:54:06,236 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/737cada8ecb6489d9f97c9213eb195a2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/737cada8ecb6489d9f97c9213eb195a2 2023-04-28 08:54:06,237 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/066dcde9580840ee8f05a63c3891cddc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/066dcde9580840ee8f05a63c3891cddc 2023-04-28 08:54:06,238 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cd1f87e73d754d1eae393d996f95f954 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/cd1f87e73d754d1eae393d996f95f954 2023-04-28 08:54:06,239 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f30839cd7df748c1a12abd9f265deed4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/f30839cd7df748c1a12abd9f265deed4 2023-04-28 08:54:06,240 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/823900a72c794ceb88e55a0b82972af1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/823900a72c794ceb88e55a0b82972af1 2023-04-28 08:54:06,267 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0c96ddf530424995b6ccb78b33e6fd44, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/187bb1f5e059489599aaef3eb180e649, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/03ba4d03b92c4f13a38d98946fd1c523, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/5245b57038024a18bfe60af8450b59df, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e6e4372cfb8d439eb0c2b82d439611c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/75730495d9544463a37fd901e88d204d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0fed88a0f2074303913b959a0d301671, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/809bd19ab67c4d4b8b626f4bc84e0eab, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/7fdf66c6820841ec8d936b623b3409db, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e38b94edde5f4570b6fa7289eb535253, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/063357517e71458cb5b24654d41aac4f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c5b4f8e356554912af433636d1c4995c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0d89d62784914232ba5e322ff334e98f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/2bdf7083765843fcbd7f9baf73d60403, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bf30db0a5f784bcc9d85ea3de7c5edad, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ce8e2213258b48538bb13b3881044593, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9305c385897d41719d8aa1a0c5f61bf1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bced7928de3442fc82a9dcae455393c6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/1fa14733c41a403ca52b1c8458c3284d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/3ee12e619990451b95d557bdb33c36ef, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ee73c7be46d7467a9c5c917ef946b970, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a1ccf2c8ae0e45f6a8e64dc7da291b01, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0e7b3c9ac152498f841cb378951bfbed, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/911ad2431aee42cdaa27f2c7618f3fdc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/412d0f0868474f028fcce8dcfdbeace3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/43a1c0ccd34247f9af55144f1db93d61, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/b39585033a56423190e777f089b5b0fc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a26c6870da9d49e5bc3663f21fd9255e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c0c4a3b3cb664792a4690bccfd89b328, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/cd4a0dd6978e4f70b9289ea1071f2bdc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/527e648647d84f31a78331184de6c2fa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/54d054297e5c48df9f1ac2b2c9bd7295, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/be0f3dd7f05f4a05977542db33d5541f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/dc3535088096487e9a9f9ca9975720ae, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/24a4a0ace2ba4202a829758d9060cb4b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/55b2d5f08b38479b8f89e7f48e04aba9] to archive 2023-04-28 08:54:06,268 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:54:06,269 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0c96ddf530424995b6ccb78b33e6fd44 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0c96ddf530424995b6ccb78b33e6fd44 2023-04-28 08:54:06,271 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/187bb1f5e059489599aaef3eb180e649 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/187bb1f5e059489599aaef3eb180e649 2023-04-28 08:54:06,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/03ba4d03b92c4f13a38d98946fd1c523 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/03ba4d03b92c4f13a38d98946fd1c523 2023-04-28 08:54:06,272 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/5245b57038024a18bfe60af8450b59df to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/5245b57038024a18bfe60af8450b59df 2023-04-28 08:54:06,274 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e6e4372cfb8d439eb0c2b82d439611c2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e6e4372cfb8d439eb0c2b82d439611c2 2023-04-28 08:54:06,275 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/75730495d9544463a37fd901e88d204d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/75730495d9544463a37fd901e88d204d 2023-04-28 08:54:06,276 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0fed88a0f2074303913b959a0d301671 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0fed88a0f2074303913b959a0d301671 2023-04-28 08:54:06,277 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/809bd19ab67c4d4b8b626f4bc84e0eab to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/809bd19ab67c4d4b8b626f4bc84e0eab 2023-04-28 08:54:06,278 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/7fdf66c6820841ec8d936b623b3409db to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/7fdf66c6820841ec8d936b623b3409db 2023-04-28 08:54:06,279 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e38b94edde5f4570b6fa7289eb535253 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e38b94edde5f4570b6fa7289eb535253 2023-04-28 08:54:06,280 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/063357517e71458cb5b24654d41aac4f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/063357517e71458cb5b24654d41aac4f 2023-04-28 08:54:06,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 08:54:06,281 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c5b4f8e356554912af433636d1c4995c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c5b4f8e356554912af433636d1c4995c 2023-04-28 08:54:06,282 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0d89d62784914232ba5e322ff334e98f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0d89d62784914232ba5e322ff334e98f 2023-04-28 08:54:06,283 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/2bdf7083765843fcbd7f9baf73d60403 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/2bdf7083765843fcbd7f9baf73d60403 2023-04-28 08:54:06,284 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bf30db0a5f784bcc9d85ea3de7c5edad to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bf30db0a5f784bcc9d85ea3de7c5edad 2023-04-28 08:54:06,285 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ce8e2213258b48538bb13b3881044593 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ce8e2213258b48538bb13b3881044593 2023-04-28 08:54:06,286 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9305c385897d41719d8aa1a0c5f61bf1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9305c385897d41719d8aa1a0c5f61bf1 2023-04-28 08:54:06,287 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bced7928de3442fc82a9dcae455393c6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/bced7928de3442fc82a9dcae455393c6 2023-04-28 08:54:06,288 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/1fa14733c41a403ca52b1c8458c3284d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/1fa14733c41a403ca52b1c8458c3284d 2023-04-28 08:54:06,289 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/3ee12e619990451b95d557bdb33c36ef to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/3ee12e619990451b95d557bdb33c36ef 2023-04-28 08:54:06,290 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ee73c7be46d7467a9c5c917ef946b970 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/ee73c7be46d7467a9c5c917ef946b970 2023-04-28 08:54:06,292 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a1ccf2c8ae0e45f6a8e64dc7da291b01 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a1ccf2c8ae0e45f6a8e64dc7da291b01 2023-04-28 08:54:06,293 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0e7b3c9ac152498f841cb378951bfbed to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/0e7b3c9ac152498f841cb378951bfbed 2023-04-28 08:54:06,294 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/911ad2431aee42cdaa27f2c7618f3fdc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/911ad2431aee42cdaa27f2c7618f3fdc 2023-04-28 08:54:06,295 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/412d0f0868474f028fcce8dcfdbeace3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/412d0f0868474f028fcce8dcfdbeace3 2023-04-28 08:54:06,297 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/43a1c0ccd34247f9af55144f1db93d61 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/43a1c0ccd34247f9af55144f1db93d61 2023-04-28 08:54:06,298 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/b39585033a56423190e777f089b5b0fc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/b39585033a56423190e777f089b5b0fc 2023-04-28 08:54:06,299 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a26c6870da9d49e5bc3663f21fd9255e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/a26c6870da9d49e5bc3663f21fd9255e 2023-04-28 08:54:06,300 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c0c4a3b3cb664792a4690bccfd89b328 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/c0c4a3b3cb664792a4690bccfd89b328 2023-04-28 08:54:06,301 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/cd4a0dd6978e4f70b9289ea1071f2bdc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/cd4a0dd6978e4f70b9289ea1071f2bdc 2023-04-28 08:54:06,303 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/527e648647d84f31a78331184de6c2fa to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/527e648647d84f31a78331184de6c2fa 2023-04-28 08:54:06,304 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/54d054297e5c48df9f1ac2b2c9bd7295 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/54d054297e5c48df9f1ac2b2c9bd7295 2023-04-28 08:54:06,305 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/be0f3dd7f05f4a05977542db33d5541f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/be0f3dd7f05f4a05977542db33d5541f 2023-04-28 08:54:06,306 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/dc3535088096487e9a9f9ca9975720ae to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/dc3535088096487e9a9f9ca9975720ae 2023-04-28 08:54:06,307 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/24a4a0ace2ba4202a829758d9060cb4b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/24a4a0ace2ba4202a829758d9060cb4b 2023-04-28 08:54:06,308 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/55b2d5f08b38479b8f89e7f48e04aba9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/55b2d5f08b38479b8f89e7f48e04aba9 2023-04-28 08:54:06,312 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9f64ded55ee64abfb740ed9246556efb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9eaa88499e0a41128b112ec7675f277d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28752df514fd41cb80173dde7a23cb25, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8b35c53f1c2e43fb9d96e3673b1a1a51, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d9c70a9cef9041caabd24c6c5764e111, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/2d5faae90a6343d9ba8bfb92876399be, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1a3b4362c4324c1db12393773f8d3bb9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8efed1ad169d4d44974526ae97f28c59, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/df9223f36f644aeb860dbba365f5bb4b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/da22208af7bb4f7991bc0f4656909401, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0dd98b8f3a64455ea80838a55dfa1e60, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6984aaf45cf34f5580f2c4d326a06558, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/5c8b3bb8106f442eac3ed0bc8fb04f24, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/85b8fbffe0354c3282b5a0e00919c369, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6122f45e3b3a42ed9d573b045fd80b20, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/bbb9435333094166b0aad8208929bb51, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9b4196688cfd4904b1c81322edfe0417, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/baf672379c864b77b0a1e9d1552ee2d2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c18435c8410a4ec481d897ba7ecfaf0c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/f7dfd1c1850e4978a906ba23b7bb982c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c75d0a6318924128afd70f9a7d439d84, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/83d2fe181f4049e89f07c4e7f11bcd3c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e8a67cdc46734cc9a05dd40951fba8c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4cc5993d156f4a73b9ea7829918f797d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c33f60da6731416296f2f756bfccfc3b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e51c58cd086f4dabbced186e11678420, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1fe51778ab5344ad8c519aedb7bcd4a6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/04b3e78df2da46e2a5048568b28fa6d0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28bf0165c59e49c59e029cf2575f3e79, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/43a9f50f78804286a6244304ef5632c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0f962fe8b23349279ef379cea87557cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d256c9fe65f84120b241578def624e77, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4c042dae1a6e463eac952fee38e904f1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4e96511b5c1c4960ae987de08c1b4fcf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4f4e2f53bb0c4aac8915fd2917543490] to archive 2023-04-28 08:54:06,313 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:54:06,314 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9f64ded55ee64abfb740ed9246556efb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9f64ded55ee64abfb740ed9246556efb 2023-04-28 08:54:06,315 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9eaa88499e0a41128b112ec7675f277d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9eaa88499e0a41128b112ec7675f277d 2023-04-28 08:54:06,316 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28752df514fd41cb80173dde7a23cb25 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28752df514fd41cb80173dde7a23cb25 2023-04-28 08:54:06,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8b35c53f1c2e43fb9d96e3673b1a1a51 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8b35c53f1c2e43fb9d96e3673b1a1a51 2023-04-28 08:54:06,318 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d9c70a9cef9041caabd24c6c5764e111 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d9c70a9cef9041caabd24c6c5764e111 2023-04-28 08:54:06,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/2d5faae90a6343d9ba8bfb92876399be to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/2d5faae90a6343d9ba8bfb92876399be 2023-04-28 08:54:06,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1a3b4362c4324c1db12393773f8d3bb9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1a3b4362c4324c1db12393773f8d3bb9 2023-04-28 08:54:06,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8efed1ad169d4d44974526ae97f28c59 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/8efed1ad169d4d44974526ae97f28c59 2023-04-28 08:54:06,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/df9223f36f644aeb860dbba365f5bb4b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/df9223f36f644aeb860dbba365f5bb4b 2023-04-28 08:54:06,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/da22208af7bb4f7991bc0f4656909401 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/da22208af7bb4f7991bc0f4656909401 2023-04-28 08:54:06,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0dd98b8f3a64455ea80838a55dfa1e60 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0dd98b8f3a64455ea80838a55dfa1e60 2023-04-28 08:54:06,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6984aaf45cf34f5580f2c4d326a06558 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6984aaf45cf34f5580f2c4d326a06558 2023-04-28 08:54:06,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/5c8b3bb8106f442eac3ed0bc8fb04f24 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/5c8b3bb8106f442eac3ed0bc8fb04f24 2023-04-28 08:54:06,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/85b8fbffe0354c3282b5a0e00919c369 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/85b8fbffe0354c3282b5a0e00919c369 2023-04-28 08:54:06,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6122f45e3b3a42ed9d573b045fd80b20 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/6122f45e3b3a42ed9d573b045fd80b20 2023-04-28 08:54:06,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/bbb9435333094166b0aad8208929bb51 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/bbb9435333094166b0aad8208929bb51 2023-04-28 08:54:06,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9b4196688cfd4904b1c81322edfe0417 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/9b4196688cfd4904b1c81322edfe0417 2023-04-28 08:54:06,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/baf672379c864b77b0a1e9d1552ee2d2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/baf672379c864b77b0a1e9d1552ee2d2 2023-04-28 08:54:06,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c18435c8410a4ec481d897ba7ecfaf0c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c18435c8410a4ec481d897ba7ecfaf0c 2023-04-28 08:54:06,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/f7dfd1c1850e4978a906ba23b7bb982c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/f7dfd1c1850e4978a906ba23b7bb982c 2023-04-28 08:54:06,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c75d0a6318924128afd70f9a7d439d84 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c75d0a6318924128afd70f9a7d439d84 2023-04-28 08:54:06,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/83d2fe181f4049e89f07c4e7f11bcd3c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/83d2fe181f4049e89f07c4e7f11bcd3c 2023-04-28 08:54:06,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e8a67cdc46734cc9a05dd40951fba8c2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e8a67cdc46734cc9a05dd40951fba8c2 2023-04-28 08:54:06,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4cc5993d156f4a73b9ea7829918f797d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4cc5993d156f4a73b9ea7829918f797d 2023-04-28 08:54:06,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c33f60da6731416296f2f756bfccfc3b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c33f60da6731416296f2f756bfccfc3b 2023-04-28 08:54:06,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e51c58cd086f4dabbced186e11678420 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/e51c58cd086f4dabbced186e11678420 2023-04-28 08:54:06,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1fe51778ab5344ad8c519aedb7bcd4a6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/1fe51778ab5344ad8c519aedb7bcd4a6 2023-04-28 08:54:06,342 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/04b3e78df2da46e2a5048568b28fa6d0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/04b3e78df2da46e2a5048568b28fa6d0 2023-04-28 08:54:06,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28bf0165c59e49c59e029cf2575f3e79 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/28bf0165c59e49c59e029cf2575f3e79 2023-04-28 08:54:06,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/43a9f50f78804286a6244304ef5632c2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/43a9f50f78804286a6244304ef5632c2 2023-04-28 08:54:06,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0f962fe8b23349279ef379cea87557cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/0f962fe8b23349279ef379cea87557cb 2023-04-28 08:54:06,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d256c9fe65f84120b241578def624e77 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d256c9fe65f84120b241578def624e77 2023-04-28 08:54:06,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4c042dae1a6e463eac952fee38e904f1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4c042dae1a6e463eac952fee38e904f1 2023-04-28 08:54:06,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4e96511b5c1c4960ae987de08c1b4fcf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4e96511b5c1c4960ae987de08c1b4fcf 2023-04-28 08:54:06,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4f4e2f53bb0c4aac8915fd2917543490 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/4f4e2f53bb0c4aac8915fd2917543490 2023-04-28 08:54:06,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/recovered.edits/628.seqid, newMaxSeqId=628, maxSeqId=1 2023-04-28 08:54:06,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796. 2023-04-28 08:54:06,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 6dfeafaee81d2f031bddf09740078796: 2023-04-28 08:54:06,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:06,357 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6dfeafaee81d2f031bddf09740078796, regionState=CLOSED 2023-04-28 08:54:06,357 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672046357"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672046357"}]},"ts":"1682672046357"} 2023-04-28 08:54:06,360 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-28 08:54:06,360 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 6dfeafaee81d2f031bddf09740078796, server=jenkins-hbase6.apache.org,40909,1682672018517 in 2.1740 sec 2023-04-28 08:54:06,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-28 08:54:06,361 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dfeafaee81d2f031bddf09740078796, UNASSIGN in 2.1800 sec 2023-04-28 08:54:06,362 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672046362"}]},"ts":"1682672046362"} 2023-04-28 08:54:06,363 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 08:54:06,365 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 08:54:06,367 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1980 sec 2023-04-28 08:54:08,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-28 08:54:08,282 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-28 08:54:08,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete TestAcidGuarantees 2023-04-28 08:54:08,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:08,295 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:08,296 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:08,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 08:54:08,301 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:08,304 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/recovered.edits] 2023-04-28 08:54:08,306 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/0c473ba1e0bd4516944a42600903cd09 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/0c473ba1e0bd4516944a42600903cd09 2023-04-28 08:54:08,308 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4b8f3f7e098a4f9e9566992b066a6378 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/4b8f3f7e098a4f9e9566992b066a6378 2023-04-28 08:54:08,309 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7600b6424485494683fe900fd68e834c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/7600b6424485494683fe900fd68e834c 2023-04-28 08:54:08,310 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ee2a087bbd3e4f4c846cf8fac2719966 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/A/ee2a087bbd3e4f4c846cf8fac2719966 2023-04-28 08:54:08,313 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/883b57cc4d3c4ff4abf5e07afe51ae81 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/883b57cc4d3c4ff4abf5e07afe51ae81 2023-04-28 08:54:08,314 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9afc635074f24b83b99ec062f71ceefe to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9afc635074f24b83b99ec062f71ceefe 2023-04-28 08:54:08,315 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9e41716579d84771afd052c973262dc2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/9e41716579d84771afd052c973262dc2 2023-04-28 08:54:08,316 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e51be93a91474d618f5ef51e83e2aa8b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/B/e51be93a91474d618f5ef51e83e2aa8b 2023-04-28 08:54:08,319 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/14a5f2ecd6414b9ba8a58d81f660a761 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/14a5f2ecd6414b9ba8a58d81f660a761 2023-04-28 08:54:08,320 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c6f1df8ee0404bf1939ff4595c79f993 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c6f1df8ee0404bf1939ff4595c79f993 2023-04-28 08:54:08,322 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c8907768855b4402bb867677e861a7ca to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/c8907768855b4402bb867677e861a7ca 2023-04-28 08:54:08,323 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d78c0ec86c8a4c6abe0f4de1250c98c8 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/C/d78c0ec86c8a4c6abe0f4de1250c98c8 2023-04-28 08:54:08,326 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/recovered.edits/628.seqid to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796/recovered.edits/628.seqid 2023-04-28 08:54:08,327 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/6dfeafaee81d2f031bddf09740078796 2023-04-28 08:54:08,327 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:54:08,329 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:08,350 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 08:54:08,393 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 08:54:08,394 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:08,394 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 08:54:08,394 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682672048394"}]},"ts":"9223372036854775807"} 2023-04-28 08:54:08,397 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 08:54:08,397 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6dfeafaee81d2f031bddf09740078796, NAME => 'TestAcidGuarantees,,1682672021788.6dfeafaee81d2f031bddf09740078796.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 08:54:08,397 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 08:54:08,397 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682672048397"}]},"ts":"9223372036854775807"} 2023-04-28 08:54:08,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 08:54:08,399 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 08:54:08,401 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:08,402 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 114 msec 2023-04-28 08:54:08,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-28 08:54:08,600 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-28 08:54:08,617 INFO [Listener at localhost/39717] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1582761948_15 at /127.0.0.1:47888 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52f5cfe6-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase6:40909-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1582761948_15 at /127.0.0.1:47954 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52f5cfe6-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52f5cfe6-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x52f5cfe6-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=593 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=469 (was 336) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=6241 (was 7016) 2023-04-28 08:54:08,631 INFO [Listener at localhost/39717] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=593, MaxFileDescriptor=60000, SystemLoadAverage=469, ProcessCount=172, AvailableMemoryMB=6240 2023-04-28 08:54:08,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:54:08,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:54:08,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:08,637 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 08:54:08,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-28 08:54:08,637 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 08:54:08,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 08:54:08,639 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:08,640 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 empty. 2023-04-28 08:54:08,640 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:08,640 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:54:08,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 08:54:08,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 08:54:09,049 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:54:09,051 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e4dc6da0d5be68cdd66fb19ccfc1d9f0, NAME => 'TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp 2023-04-28 08:54:09,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 08:54:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:54:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e4dc6da0d5be68cdd66fb19ccfc1d9f0, disabling compactions & flushes 2023-04-28 08:54:09,458 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. after waiting 0 ms 2023-04-28 08:54:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,458 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,458 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:09,461 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 08:54:09,462 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672049461"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672049461"}]},"ts":"1682672049461"} 2023-04-28 08:54:09,463 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 08:54:09,464 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 08:54:09,465 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672049464"}]},"ts":"1682672049464"} 2023-04-28 08:54:09,466 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 08:54:09,469 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, ASSIGN}] 2023-04-28 08:54:09,471 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, ASSIGN 2023-04-28 08:54:09,472 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=false 2023-04-28 08:54:09,623 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:09,623 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672049623"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672049623"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672049623"}]},"ts":"1682672049623"} 2023-04-28 08:54:09,625 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:54:09,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 08:54:09,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e4dc6da0d5be68cdd66fb19ccfc1d9f0, NAME => 'TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:54:09,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:54:09,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,782 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,784 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,785 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A 2023-04-28 08:54:09,785 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A 2023-04-28 08:54:09,785 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:09,786 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e4dc6da0d5be68cdd66fb19ccfc1d9f0 columnFamilyName A 2023-04-28 08:54:09,786 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(310): Store=e4dc6da0d5be68cdd66fb19ccfc1d9f0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:09,786 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,787 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B 2023-04-28 08:54:09,788 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B 2023-04-28 08:54:09,788 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:09,788 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e4dc6da0d5be68cdd66fb19ccfc1d9f0 columnFamilyName B 2023-04-28 08:54:09,788 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(310): Store=e4dc6da0d5be68cdd66fb19ccfc1d9f0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:09,789 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,790 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C 2023-04-28 08:54:09,790 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C 2023-04-28 08:54:09,790 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:09,790 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e4dc6da0d5be68cdd66fb19ccfc1d9f0 columnFamilyName C 2023-04-28 08:54:09,791 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(310): Store=e4dc6da0d5be68cdd66fb19ccfc1d9f0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:09,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:54:09,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:09,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:54:09,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e4dc6da0d5be68cdd66fb19ccfc1d9f0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64668476, jitterRate=-0.036364614963531494}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:54:09,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:09,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., pid=18, masterSystemTime=1682672049777 2023-04-28 08:54:09,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:09,799 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:09,800 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672049799"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672049799"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672049799"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672049799"}]},"ts":"1682672049799"} 2023-04-28 08:54:09,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-28 08:54:09,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 in 176 msec 2023-04-28 08:54:09,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-28 08:54:09,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, ASSIGN in 333 msec 2023-04-28 08:54:09,805 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 08:54:09,805 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672049805"}]},"ts":"1682672049805"} 2023-04-28 08:54:09,806 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 08:54:09,808 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 08:54:09,809 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1740 sec 2023-04-28 08:54:10,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-28 08:54:10,744 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-28 08:54:10,745 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x7829b43e to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:10,750 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@186ae08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:10,752 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:10,756 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:10,758 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 08:54:10,761 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:38636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 08:54:10,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:54:10,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$12(2454): Client=jenkins//172.31.2.20 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:54:10,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:11,196 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 08:54:11,197 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:54:11,197 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 08:54:11,202 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 08:54:11,214 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, REOPEN/MOVE}] 2023-04-28 08:54:11,215 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, REOPEN/MOVE 2023-04-28 08:54:11,216 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,216 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672051216"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672051216"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672051216"}]},"ts":"1682672051216"} 2023-04-28 08:54:11,217 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:54:11,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e4dc6da0d5be68cdd66fb19ccfc1d9f0, disabling compactions & flushes 2023-04-28 08:54:11,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. after waiting 0 ms 2023-04-28 08:54:11,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 08:54:11,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,376 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:11,376 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(3509): Not adding moved region record: e4dc6da0d5be68cdd66fb19ccfc1d9f0 to self. 2023-04-28 08:54:11,378 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,378 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=CLOSED 2023-04-28 08:54:11,378 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672051378"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672051378"}]},"ts":"1682672051378"} 2023-04-28 08:54:11,381 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-28 08:54:11,381 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 in 163 msec 2023-04-28 08:54:11,382 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=true 2023-04-28 08:54:11,532 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,533 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672051532"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672051532"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672051532"}]},"ts":"1682672051532"} 2023-04-28 08:54:11,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:54:11,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e4dc6da0d5be68cdd66fb19ccfc1d9f0, NAME => 'TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:54:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:54:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,693 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,694 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A 2023-04-28 08:54:11,694 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A 2023-04-28 08:54:11,694 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:11,697 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e4dc6da0d5be68cdd66fb19ccfc1d9f0 columnFamilyName A 2023-04-28 08:54:11,699 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(310): Store=e4dc6da0d5be68cdd66fb19ccfc1d9f0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:11,699 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,701 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B 2023-04-28 08:54:11,701 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B 2023-04-28 08:54:11,701 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:11,701 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e4dc6da0d5be68cdd66fb19ccfc1d9f0 columnFamilyName B 2023-04-28 08:54:11,702 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(310): Store=e4dc6da0d5be68cdd66fb19ccfc1d9f0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:11,702 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,703 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C 2023-04-28 08:54:11,703 DEBUG [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C 2023-04-28 08:54:11,703 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:11,703 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e4dc6da0d5be68cdd66fb19ccfc1d9f0 columnFamilyName C 2023-04-28 08:54:11,704 INFO [StoreOpener-e4dc6da0d5be68cdd66fb19ccfc1d9f0-1] regionserver.HStore(310): Store=e4dc6da0d5be68cdd66fb19ccfc1d9f0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:11,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:54:11,709 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened e4dc6da0d5be68cdd66fb19ccfc1d9f0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59861088, jitterRate=-0.10800027847290039}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:54:11,710 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:11,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., pid=23, masterSystemTime=1682672051686 2023-04-28 08:54:11,713 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,713 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,713 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672051713"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672051713"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672051713"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672051713"}]},"ts":"1682672051713"} 2023-04-28 08:54:11,716 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-28 08:54:11,716 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 in 181 msec 2023-04-28 08:54:11,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-28 08:54:11,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, REOPEN/MOVE in 502 msec 2023-04-28 08:54:11,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-28 08:54:11,722 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 519 msec 2023-04-28 08:54:11,724 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 948 msec 2023-04-28 08:54:11,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-28 08:54:11,733 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x21281a37 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,738 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55cbb115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,739 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x1441c9f0 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,744 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21265fcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,744 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x73e51571 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,748 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69f15cb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,749 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x6c1c92c6 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,755 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@feb770d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,756 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5c472cfd to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,761 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54451f11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,762 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x2c4379c2 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,767 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@508246cc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,767 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x3f43b712 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,771 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8795451, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,772 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x7a3bb940 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,777 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58e27a32, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,777 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x2efdb02f to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:11,782 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@92bed09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:11,788 DEBUG [hconnection-0x28f5e7f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,788 DEBUG [hconnection-0x78c30c21-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,789 DEBUG [hconnection-0x36864547-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,792 DEBUG [hconnection-0x6552908d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,792 DEBUG [hconnection-0x7d198ced-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,794 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,798 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,798 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,799 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58010, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,799 DEBUG [hconnection-0x475c8d20-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,799 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,800 DEBUG [hconnection-0x4a3705ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,800 DEBUG [hconnection-0x485e094d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,802 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,810 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,810 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,822 DEBUG [hconnection-0x7f5a7560-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:11,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:11,841 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:58078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:11,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:11,858 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:11,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:11,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:11,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:11,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:11,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:11,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:11,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:11,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:11,859 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:11,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:11,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:11,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:11,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:11,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:11,864 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:11,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:11,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:11,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:11,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:11,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:11,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:11,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:11,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:11,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:11,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:11,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:11,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:11,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:11,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:11,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:11,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:11,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:11,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,888 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 08:54:11,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672111887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672111887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672111889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672111891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672111889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:11,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:11,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:11,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:11,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:11,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46d329bc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:11,900 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:11,900 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,903 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,903 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:11,903 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:11,903 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:11,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:11,905 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:11,905 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,905 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:11,905 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:11,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:11,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:11,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:11,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:11,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:11,910 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:11,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:11,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:11,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:11,910 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:11,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:11,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:11,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:11,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:11,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,920 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,936 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:11,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:11,937 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:11,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:11,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:11,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:11,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:11,937 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:11,937 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:11,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:11,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672111993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672111993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672111994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,996 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:11,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672111995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:11,996 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a88f839aa840423fb218c23fadded6b6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a88f839aa840423fb218c23fadded6b6 2023-04-28 08:54:11,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672111995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,017 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/08cfbaa077c44b63af2c87497fe1fa51 2023-04-28 08:54:12,037 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:12,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:12,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:12,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:12,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:12,072 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:12,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:12,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:12,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,075 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:12,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:12,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:12,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:12,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:12,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:12,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:12,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:12,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:12,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:12,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:12,083 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:12,083 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:12,083 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:12,083 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:12,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:12,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:12,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,085 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:12,085 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:12,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:12,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1859e40[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:12,090 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:12,090 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,092 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:12,092 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:12,092 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:12,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:12,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:12,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:12,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:12,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:12,098 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:12,098 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:12,098 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:12,098 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:12,098 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:12,101 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,101 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:12,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,101 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:12,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,102 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:12,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:12,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:12,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:12,108 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:12,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:12,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:12,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:12,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:12,109 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:12,109 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:12,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:12,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672112197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672112198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672112198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672112199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672112199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,209 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:12,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:12,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:12,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:12,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:12,248 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:12,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:12,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:12,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,250 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:12,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:12,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:12,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:12,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:12,271 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:12,271 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,271 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:12,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:12,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:12,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:12,284 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:12,284 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:12,284 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:12,284 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:12,284 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:12,284 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,284 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:12,284 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:12,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,287 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,287 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:12,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:12,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aa1d32b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:12,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:12,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:12,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:12,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:12,298 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:12,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:12,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:12,301 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:12,305 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:12,305 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:12,305 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:12,305 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:12,305 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:12,307 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,307 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,307 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:12,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:12,307 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,309 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:12,319 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:12,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:12,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:12,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:12,320 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:12,320 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:12,420 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:12,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:12,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:12,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:12,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:12,445 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:12,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:12,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:12,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:12,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:12,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:12,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:12,461 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:12,461 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,461 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:12,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/3db7ba5b08e74d599ac484cd8d93b8c3 2023-04-28 08:54:12,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:12,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:12,467 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:12,467 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:12,467 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:12,467 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:12,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:12,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:12,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,475 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:12,475 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:12,477 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,478 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,478 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:12,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3305544d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:12,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:12,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:12,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:12,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:12,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:12,485 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:12,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:12,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:12,488 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:12,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:12,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:12,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:12,489 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:12,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:12,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:12,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,492 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:12,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:12,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:12,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:12,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:12,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:12,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:12,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,496 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:12,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:12,496 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:12,496 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:12,496 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:12,496 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,496 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:12,496 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672112501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672112503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672112504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672112504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:12,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672112505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/99405482cc37403989347210ddc4e24b 2023-04-28 08:54:12,519 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/08cfbaa077c44b63af2c87497fe1fa51 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/08cfbaa077c44b63af2c87497fe1fa51 2023-04-28 08:54:12,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/08cfbaa077c44b63af2c87497fe1fa51, entries=150, sequenceid=16, filesize=25.1 K 2023-04-28 08:54:12,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/3db7ba5b08e74d599ac484cd8d93b8c3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/3db7ba5b08e74d599ac484cd8d93b8c3 2023-04-28 08:54:12,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/3db7ba5b08e74d599ac484cd8d93b8c3, entries=150, sequenceid=16, filesize=11.6 K 2023-04-28 08:54:12,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/99405482cc37403989347210ddc4e24b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/99405482cc37403989347210ddc4e24b 2023-04-28 08:54:12,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/99405482cc37403989347210ddc4e24b, entries=150, sequenceid=16, filesize=11.6 K 2023-04-28 08:54:12,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 689ms, sequenceid=16, compaction requested=false 2023-04-28 08:54:12,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:12,596 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:12,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:12,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:12,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:12,603 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:12,603 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,603 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:12,603 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:12,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:12,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:12,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:12,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:12,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:12,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:12,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:12,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:12,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:12,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:12,608 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:12,608 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:12,608 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:12,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:12,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:12,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:12,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:12,611 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:12,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:12,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:12,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:12,611 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:12,612 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:12,612 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:12,646 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042886c5df9b39b941b29afe19059f891fc9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042886c5df9b39b941b29afe19059f891fc9 2023-04-28 08:54:12,662 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/ee6df7b3328542a09b44d341390a71eb 2023-04-28 08:54:12,745 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/dc4dc98b534b4f45a776150fcbc0ddfc 2023-04-28 08:54:13,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:13,008 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:13,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672113017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672113017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672113019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672113019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672113021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672113122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672113122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672113125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672113125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672113126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,166 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/7399af4a7c394cd598bd04ae3ae71102 2023-04-28 08:54:13,172 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/ee6df7b3328542a09b44d341390a71eb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ee6df7b3328542a09b44d341390a71eb 2023-04-28 08:54:13,177 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ee6df7b3328542a09b44d341390a71eb, entries=150, sequenceid=36, filesize=25.1 K 2023-04-28 08:54:13,178 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/dc4dc98b534b4f45a776150fcbc0ddfc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/dc4dc98b534b4f45a776150fcbc0ddfc 2023-04-28 08:54:13,183 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/dc4dc98b534b4f45a776150fcbc0ddfc, entries=150, sequenceid=36, filesize=11.6 K 2023-04-28 08:54:13,187 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/7399af4a7c394cd598bd04ae3ae71102 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7399af4a7c394cd598bd04ae3ae71102 2023-04-28 08:54:13,193 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7399af4a7c394cd598bd04ae3ae71102, entries=150, sequenceid=36, filesize=11.6 K 2023-04-28 08:54:13,194 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 583ms, sequenceid=36, compaction requested=false 2023-04-28 08:54:13,194 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:13,194 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:13,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:13,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:13,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:13,198 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,198 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,198 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,198 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:13,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:13,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc41558[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:13,200 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:13,200 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,201 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:13,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:13,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:13,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:13,203 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,204 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,204 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:13,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:13,207 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:13,207 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:13,207 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:13,207 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:13,207 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:13,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,215 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:13,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:13,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:13,215 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:13,215 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:13,215 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,315 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:13,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:13,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:13,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:54:13,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:13,330 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:13,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:13,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:13,335 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:13,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:13,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:13,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,338 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,338 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:13,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:13,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:13,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:13,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,355 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:13,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:13,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:13,375 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,375 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:13,376 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:13,376 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:13,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:13,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:13,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:13,383 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:13,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@411ee11e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:13,386 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:13,386 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672113377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672113379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:13,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:13,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:13,388 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,388 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:13,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:13,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:13,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672113387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672113388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672113395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:13,406 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:13,406 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:13,406 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:13,406 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:13,406 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:13,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,409 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:13,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:13,415 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,416 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:13,416 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:13,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:13,432 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428392bc96951084f5d8b0aebdd26597633 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428392bc96951084f5d8b0aebdd26597633 2023-04-28 08:54:13,467 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/2e604026032f4252a262e30313b1a062 2023-04-28 08:54:13,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672113488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672113489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672113501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672113501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672113513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,516 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:13,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:13,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:13,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:13,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:13,535 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:13,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:13,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:13,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/a7aa7c37a10343168cda27caa7b621e0 2023-04-28 08:54:13,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,537 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:13,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:13,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:13,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:13,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:13,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:13,552 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:13,552 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:13,552 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,553 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:13,553 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:13,553 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,553 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:13,553 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:13,553 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:13,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:13,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f25a9d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:13,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:13,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:13,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:13,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:13,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:13,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:13,575 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:13,578 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:13,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:13,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:13,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:13,579 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:13,580 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,580 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,580 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,581 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,582 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,586 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,587 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:13,587 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:13,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:13,612 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/4087a608bd724dc08051fb3f1f4f6e0f 2023-04-28 08:54:13,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/2e604026032f4252a262e30313b1a062 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2e604026032f4252a262e30313b1a062 2023-04-28 08:54:13,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2e604026032f4252a262e30313b1a062, entries=150, sequenceid=50, filesize=25.1 K 2023-04-28 08:54:13,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/a7aa7c37a10343168cda27caa7b621e0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7aa7c37a10343168cda27caa7b621e0 2023-04-28 08:54:13,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7aa7c37a10343168cda27caa7b621e0, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 08:54:13,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/4087a608bd724dc08051fb3f1f4f6e0f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/4087a608bd724dc08051fb3f1f4f6e0f 2023-04-28 08:54:13,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/4087a608bd724dc08051fb3f1f4f6e0f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 08:54:13,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 332ms, sequenceid=50, compaction requested=true 2023-04-28 08:54:13,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:13,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:13,659 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:13,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:13,659 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:13,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:13,661 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:13,661 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:13,661 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/A is initiating minor compaction (all files) 2023-04-28 08:54:13,661 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/B is initiating minor compaction (all files) 2023-04-28 08:54:13,661 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/A in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,661 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/08cfbaa077c44b63af2c87497fe1fa51, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ee6df7b3328542a09b44d341390a71eb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2e604026032f4252a262e30313b1a062] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=75.4 K 2023-04-28 08:54:13,661 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/B in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,662 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/3db7ba5b08e74d599ac484cd8d93b8c3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/dc4dc98b534b4f45a776150fcbc0ddfc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7aa7c37a10343168cda27caa7b621e0] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=34.8 K 2023-04-28 08:54:13,662 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 08cfbaa077c44b63af2c87497fe1fa51, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682672051842 2023-04-28 08:54:13,662 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3db7ba5b08e74d599ac484cd8d93b8c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682672051842 2023-04-28 08:54:13,662 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ee6df7b3328542a09b44d341390a71eb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682672051886 2023-04-28 08:54:13,663 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dc4dc98b534b4f45a776150fcbc0ddfc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682672051886 2023-04-28 08:54:13,663 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 2e604026032f4252a262e30313b1a062, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672053019 2023-04-28 08:54:13,663 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a7aa7c37a10343168cda27caa7b621e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672053019 2023-04-28 08:54:13,687 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:13,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:13,692 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#B#compaction#128 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:13,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:13,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:13,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:13,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:13,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:13,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:13,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:13,698 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:13,698 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:13,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:13,701 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:13,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:13,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:13,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:13,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:13,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672113714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672113714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672113716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672113717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672113717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:13,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:13,735 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:13,735 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,735 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:13,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:13,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:13,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:13,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:13,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:13,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:13,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,765 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#A#compaction#129 average throughput is 0.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:13,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:13,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:13,776 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,777 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,778 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cccd352[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:13,782 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:13,782 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:13,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:13,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:13,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:13,792 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,792 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:13,793 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,793 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:13,796 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:13,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:13,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:13,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:13,796 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:13,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,800 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:13,808 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:13,808 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:13,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:13,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672113824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672113824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:13,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672113824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672113825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,908 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:13,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:13,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:13,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:13,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:13,914 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:13,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:13,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:13,915 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,915 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,916 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,916 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,917 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,917 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:13,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:13,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:13,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:13,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:13,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:13,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:13,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:13,918 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,918 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:13,918 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:13,918 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:13,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:13,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:13,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:13,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:13,922 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,922 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,923 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,923 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:13,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406e6946[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:13,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:13,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,931 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:13,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:13,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:13,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:13,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:13,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:13,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:13,935 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:13,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:13,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:13,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:13,936 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:13,938 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:13,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:13,941 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:13,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:13,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:13,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:13,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:13,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:13,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:13,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:13,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:13,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:13,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:13,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:13,960 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:13,960 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:13,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:14,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672114025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672114031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672114032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672114033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672114034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,060 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:14,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:14,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:14,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:14,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:14,064 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:14,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:14,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:14,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,065 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:14,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:14,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:14,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:14,068 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,068 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:14,068 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:14,068 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:14,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:14,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,070 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,070 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,070 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:14,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:14,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76ce7644[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:14,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:14,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,073 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:14,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:14,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:14,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:14,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:14,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:14,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:14,085 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:14,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:14,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:14,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:14,085 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:14,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,088 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,090 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:14,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:14,093 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:14,094 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:14,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:14,145 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/2597f746b83c4357af9062ee7025013d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2597f746b83c4357af9062ee7025013d 2023-04-28 08:54:14,155 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/B of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 2597f746b83c4357af9062ee7025013d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:14,155 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:14,155 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/B, priority=13, startTime=1682672053659; duration=0sec 2023-04-28 08:54:14,155 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:14,155 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:14,156 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:14,157 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/C is initiating minor compaction (all files) 2023-04-28 08:54:14,157 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/C in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,157 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/99405482cc37403989347210ddc4e24b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7399af4a7c394cd598bd04ae3ae71102, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/4087a608bd724dc08051fb3f1f4f6e0f] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=34.8 K 2023-04-28 08:54:14,157 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 99405482cc37403989347210ddc4e24b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1682672051842 2023-04-28 08:54:14,158 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7399af4a7c394cd598bd04ae3ae71102, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1682672051886 2023-04-28 08:54:14,159 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4087a608bd724dc08051fb3f1f4f6e0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672053019 2023-04-28 08:54:14,166 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#C#compaction#131 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:14,183 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/64a461af211c4f9ba28a54eed29b4f1e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/64a461af211c4f9ba28a54eed29b4f1e 2023-04-28 08:54:14,189 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/C of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 64a461af211c4f9ba28a54eed29b4f1e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:14,189 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:14,189 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/C, priority=13, startTime=1682672053659; duration=0sec 2023-04-28 08:54:14,189 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:14,194 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:14,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:14,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:14,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:14,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:14,199 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:14,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:14,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:14,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,201 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,201 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:14,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:14,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:14,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:14,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:14,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:14,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:14,207 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,207 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:14,207 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:14,207 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:14,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:14,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:14,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:14,210 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,210 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:14,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71efecb2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:14,214 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:14,214 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,215 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:14,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:14,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:14,216 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428aa6800cbf1a14bedb67d0f5a042bd58d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aa6800cbf1a14bedb67d0f5a042bd58d 2023-04-28 08:54:14,216 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,216 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:14,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:14,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:14,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:14,221 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:14,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:14,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:14,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:14,221 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:14,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,225 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,228 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/a85a15304ce54856871b85bd262ffdd1 2023-04-28 08:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,231 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,231 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,231 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:14,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:14,230 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,231 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:14,231 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:14,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:14,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/f4253a3147a346bca87868e82ef2c0e0 2023-04-28 08:54:14,331 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:14,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:14,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:14,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:14,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:14,336 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:14,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:14,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:14,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672114334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,339 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:14,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672114336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672114336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:14,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:14,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:14,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:14,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:14,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672114340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:14,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:14,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:14,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:14,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:14,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:14,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:14,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:14,345 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,345 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,346 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,346 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:14,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bd3f79e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:14,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:14,348 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:14,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:14,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:14,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:14,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:14,353 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:14,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:14,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:14,357 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:14,357 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:14,357 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:14,357 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:14,358 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:14,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,361 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,365 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,368 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,368 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,368 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,368 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,369 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,369 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,369 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,369 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:14,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:14,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,371 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:14,371 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:14,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,471 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:14,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:14,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:14,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:14,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:14,486 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:14,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:14,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:14,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,488 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:14,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:14,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:14,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:14,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:14,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:14,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:14,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:14,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:14,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:14,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@284489fb[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:14,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:14,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,497 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,497 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:14,498 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:14,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:14,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:14,499 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,499 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:14,500 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,500 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:14,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:14,504 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:14,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:14,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:14,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:14,504 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:14,515 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,515 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,515 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:14,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:14,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:14,523 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:14,523 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:14,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:14,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672114534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,623 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:14,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:14,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:14,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:14,628 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:14,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:14,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:14,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:14,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:14,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:14,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:14,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:14,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:14,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:14,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:14,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:14,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:14,633 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:14,633 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:14,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:14,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:14,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:14,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,635 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,635 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:14,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d3db6e1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:14,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:14,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,643 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:14,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:14,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:14,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:14,645 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:14,648 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:14,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:14,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:14,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:14,648 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:14,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:14,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:14,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:14,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:14,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:14,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,662 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,662 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,662 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,662 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:14,662 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,662 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:14,662 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:14,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:14,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:14,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:14,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/744edc0fe207438a84a02f3ad6b4949a 2023-04-28 08:54:14,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/a85a15304ce54856871b85bd262ffdd1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/a85a15304ce54856871b85bd262ffdd1 2023-04-28 08:54:14,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/a85a15304ce54856871b85bd262ffdd1, entries=200, sequenceid=70, filesize=31.9 K 2023-04-28 08:54:14,682 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/f4253a3147a346bca87868e82ef2c0e0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f4253a3147a346bca87868e82ef2c0e0 2023-04-28 08:54:14,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f4253a3147a346bca87868e82ef2c0e0, entries=150, sequenceid=70, filesize=11.6 K 2023-04-28 08:54:14,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/744edc0fe207438a84a02f3ad6b4949a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/744edc0fe207438a84a02f3ad6b4949a 2023-04-28 08:54:14,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/744edc0fe207438a84a02f3ad6b4949a, entries=150, sequenceid=70, filesize=11.6 K 2023-04-28 08:54:14,696 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/8ac31ee9736248519f6276fc744a515c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8ac31ee9736248519f6276fc744a515c 2023-04-28 08:54:14,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 1000ms, sequenceid=70, compaction requested=false 2023-04-28 08:54:14,697 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:14,702 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/A of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 8ac31ee9736248519f6276fc744a515c(size=25.2 K), total size for store is 57.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 08:54:14,702 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:14,702 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/A, priority=13, startTime=1682672053659; duration=1sec 2023-04-28 08:54:14,702 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:14,762 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:14,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:14,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:14,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:14,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:14,779 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:14,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:14,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:14,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:14,782 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:14,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:14,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:14,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:14,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:14,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:14,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:14,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:14,819 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:14,819 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:14,819 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:14,819 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:14,819 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:14,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:14,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:14,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:14,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:14,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:14,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:14,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:14,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:14,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672114899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672114899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:14,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672114901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:14,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672114901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672115004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672115004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672115008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672115008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672115206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672115206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672115211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672115212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,237 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304280388f7e13a954c3cbb22efc858aa000a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280388f7e13a954c3cbb22efc858aa000a 2023-04-28 08:54:15,248 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/8baca76ca8e74d2e9c185c6497e71b66 2023-04-28 08:54:15,279 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/0f10c83d7077409783a99b6cd165256e 2023-04-28 08:54:15,299 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/b6e710ae44984fe2b30119e4ab5aef41 2023-04-28 08:54:15,310 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/8baca76ca8e74d2e9c185c6497e71b66 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8baca76ca8e74d2e9c185c6497e71b66 2023-04-28 08:54:15,319 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8baca76ca8e74d2e9c185c6497e71b66, entries=150, sequenceid=85, filesize=25.1 K 2023-04-28 08:54:15,320 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/0f10c83d7077409783a99b6cd165256e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0f10c83d7077409783a99b6cd165256e 2023-04-28 08:54:15,329 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0f10c83d7077409783a99b6cd165256e, entries=150, sequenceid=85, filesize=11.6 K 2023-04-28 08:54:15,330 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/b6e710ae44984fe2b30119e4ab5aef41 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b6e710ae44984fe2b30119e4ab5aef41 2023-04-28 08:54:15,339 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b6e710ae44984fe2b30119e4ab5aef41, entries=150, sequenceid=85, filesize=11.6 K 2023-04-28 08:54:15,340 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 521ms, sequenceid=85, compaction requested=true 2023-04-28 08:54:15,341 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:15,341 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:15,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:15,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:15,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:15,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:15,345 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,345 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:15,345 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:15,345 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:15,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:15,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b95588[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:15,348 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:15,348 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,349 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:15,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:15,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:15,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:15,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:15,351 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:15,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:15,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:15,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:15,354 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:15,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:15,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:15,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:15,354 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:15,356 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,356 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:15,357 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:15,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:15,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:15,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:15,359 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:15,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:15,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:15,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:15,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:15,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:15,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:15,374 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:15,374 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:15,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,374 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:15,374 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:15,374 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:15,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:15,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:15,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:15,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:15,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:15,474 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:15,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:15,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:15,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:15,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:15,478 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:15,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:15,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:15,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:15,479 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,479 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:15,480 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:15,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:15,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:15,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:15,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:15,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:15,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:15,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:15,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:15,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:15,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:15,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:15,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:15,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:15,482 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:15,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:15,483 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:15,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:15,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:15,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:15,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:15,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:15,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:15,504 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304283878f5c9f6924438972c3ed6585d10cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283878f5c9f6924438972c3ed6585d10cb 2023-04-28 08:54:15,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:15,510 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:15,519 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/86898ed11af2498eaee798968ec50cf7 2023-04-28 08:54:15,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672115520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672115520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672115522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672115523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672115540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672115624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672115624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672115625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672115627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672115828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672115828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672115828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:15,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672115830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:15,939 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/f6ba6f2ebd8846c7bc37744c5f9e72c2 2023-04-28 08:54:16,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672116130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672116131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672116131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672116132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,352 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/beb7e451c9f9459a97280486cf70b46f 2023-04-28 08:54:16,358 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/86898ed11af2498eaee798968ec50cf7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/86898ed11af2498eaee798968ec50cf7 2023-04-28 08:54:16,363 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/86898ed11af2498eaee798968ec50cf7, entries=150, sequenceid=105, filesize=25.1 K 2023-04-28 08:54:16,365 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/f6ba6f2ebd8846c7bc37744c5f9e72c2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f6ba6f2ebd8846c7bc37744c5f9e72c2 2023-04-28 08:54:16,370 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f6ba6f2ebd8846c7bc37744c5f9e72c2, entries=150, sequenceid=105, filesize=11.6 K 2023-04-28 08:54:16,371 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/beb7e451c9f9459a97280486cf70b46f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/beb7e451c9f9459a97280486cf70b46f 2023-04-28 08:54:16,376 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/beb7e451c9f9459a97280486cf70b46f, entries=150, sequenceid=105, filesize=11.6 K 2023-04-28 08:54:16,377 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 894ms, sequenceid=105, compaction requested=true 2023-04-28 08:54:16,377 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:16,377 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:16,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:16,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:16,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:16,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:16,379 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:16,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:16,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:16,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:16,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27522966[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:16,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:16,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,384 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:16,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:16,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:16,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:16,386 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:16,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:16,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:16,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:16,391 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:16,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:16,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:16,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:16,391 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:16,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:16,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:16,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:16,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:16,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:16,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:16,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:16,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,406 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:16,406 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:16,406 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:16,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:16,407 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:16,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:16,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:16,507 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:16,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:16,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:16,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:16,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:16,510 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:16,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:16,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:16,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:16,511 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,511 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:16,512 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:16,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:16,512 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:16,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:16,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:16,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:16,513 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:16,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:16,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:16,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:16,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:16,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:16,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:16,514 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:16,514 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:16,514 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:16,514 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:16,514 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:16,515 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:16,515 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:16,515 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:16,515 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:16,538 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428b9bb6bfa87b7444899d8570e249997c0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b9bb6bfa87b7444899d8570e249997c0 2023-04-28 08:54:16,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:16,636 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:16,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672116677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672116679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672116679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672116682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672116783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672116786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672116786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672116786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,878 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 08:54:16,944 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/becfd2c8fa344c8abf323e23af064750 2023-04-28 08:54:16,957 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/8221568c8b284388a023ee87a0982e6c 2023-04-28 08:54:16,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672116988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672116989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672116993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:16,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:16,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672116993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672117289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672117293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672117296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672117296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,381 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/5a55a6ad54614f80aac771b221fec013 2023-04-28 08:54:17,386 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/becfd2c8fa344c8abf323e23af064750 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/becfd2c8fa344c8abf323e23af064750 2023-04-28 08:54:17,390 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/becfd2c8fa344c8abf323e23af064750, entries=150, sequenceid=117, filesize=25.1 K 2023-04-28 08:54:17,391 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/8221568c8b284388a023ee87a0982e6c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8221568c8b284388a023ee87a0982e6c 2023-04-28 08:54:17,395 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8221568c8b284388a023ee87a0982e6c, entries=150, sequenceid=117, filesize=11.6 K 2023-04-28 08:54:17,398 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/5a55a6ad54614f80aac771b221fec013 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5a55a6ad54614f80aac771b221fec013 2023-04-28 08:54:17,402 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5a55a6ad54614f80aac771b221fec013, entries=150, sequenceid=117, filesize=11.6 K 2023-04-28 08:54:17,403 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 889ms, sequenceid=117, compaction requested=true 2023-04-28 08:54:17,403 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:17,403 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:17,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:17,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:17,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:17,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:17,405 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,405 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:17,406 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:17,406 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:17,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:17,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd57726[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:17,408 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:17,408 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:17,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:17,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:17,409 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,409 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:17,412 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,412 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:17,412 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:17,412 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:17,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:17,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:17,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:17,414 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:17,414 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:17,414 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:17,414 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:17,415 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:17,416 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,416 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:17,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,417 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:17,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:17,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:17,418 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:17,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:17,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:17,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:17,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:17,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:17,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:17,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:17,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:17,421 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,421 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,421 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:17,421 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:17,421 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:17,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:17,421 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:17,521 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:17,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:17,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:17,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:17,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:17,525 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:17,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:17,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:17,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:17,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:17,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,526 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:17,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:17,527 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,527 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:17,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:17,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:17,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:17,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:17,528 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:17,528 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:17,528 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:17,528 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:17,528 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:17,528 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:17,528 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:17,528 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:17,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:17,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:17,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:17,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:17,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:17,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:17,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:17,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:17,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672117591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672117696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672117792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672117798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672117803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672117804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:17,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672117899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:17,943 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304289d0a21cdd15442c881dacc4253ad315c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289d0a21cdd15442c881dacc4253ad315c 2023-04-28 08:54:17,950 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/35e614ab4f0b414181ac465543dafd5f 2023-04-28 08:54:17,977 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/43e060380e664c1590aee2654a9d1c53 2023-04-28 08:54:18,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:18,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672118200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,394 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/07c1dcb1570e4d2b989a113022f716b1 2023-04-28 08:54:18,400 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/35e614ab4f0b414181ac465543dafd5f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/35e614ab4f0b414181ac465543dafd5f 2023-04-28 08:54:18,404 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/35e614ab4f0b414181ac465543dafd5f, entries=150, sequenceid=137, filesize=25.3 K 2023-04-28 08:54:18,405 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/43e060380e664c1590aee2654a9d1c53 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/43e060380e664c1590aee2654a9d1c53 2023-04-28 08:54:18,411 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/43e060380e664c1590aee2654a9d1c53, entries=150, sequenceid=137, filesize=11.8 K 2023-04-28 08:54:18,412 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/07c1dcb1570e4d2b989a113022f716b1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/07c1dcb1570e4d2b989a113022f716b1 2023-04-28 08:54:18,419 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/07c1dcb1570e4d2b989a113022f716b1, entries=150, sequenceid=137, filesize=11.8 K 2023-04-28 08:54:18,420 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 892ms, sequenceid=137, compaction requested=true 2023-04-28 08:54:18,420 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:18,420 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:18,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:18,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:18,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,423 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:18,423 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:18,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:18,426 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,426 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:18,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:18,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1f5603[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:18,428 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:18,428 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,432 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:18,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:18,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:18,432 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,432 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:18,433 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:18,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:18,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:18,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:18,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:18,438 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:18,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:18,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:18,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:18,438 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:18,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:18,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:18,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:18,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:18,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:18,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:18,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:18,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:18,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:18,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:18,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:18,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:18,447 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:18,447 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:18,547 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:18,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:18,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:18,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:18,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:18,559 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:18,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:18,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:18,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:18,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:18,561 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:18,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:18,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:18,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:18,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:18,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:18,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:18,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:18,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:18,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:18,563 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:18,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:18,563 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:18,563 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:18,563 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:18,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:18,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:18,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:18,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:18,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:18,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:18,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:18,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:18,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672118798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:18,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672118799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672118808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672118810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:18,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672118812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:18,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672118906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:18,982 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428dbc0452836b84a788d3d0601267f538d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dbc0452836b84a788d3d0601267f538d 2023-04-28 08:54:19,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:19,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672119109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:19,388 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/7b71690ee46746e18f15e768500bc3f2 2023-04-28 08:54:19,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:19,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672119414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:19,799 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/510ec16b67824d459ff144f1d02e63fa 2023-04-28 08:54:19,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:19,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672119920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,210 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/e0d03acaa884442daaa8df2655c9b0f9 2023-04-28 08:54:20,216 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/7b71690ee46746e18f15e768500bc3f2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/7b71690ee46746e18f15e768500bc3f2 2023-04-28 08:54:20,221 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/7b71690ee46746e18f15e768500bc3f2, entries=150, sequenceid=149, filesize=25.3 K 2023-04-28 08:54:20,222 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/510ec16b67824d459ff144f1d02e63fa as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/510ec16b67824d459ff144f1d02e63fa 2023-04-28 08:54:20,232 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/510ec16b67824d459ff144f1d02e63fa, entries=150, sequenceid=149, filesize=11.8 K 2023-04-28 08:54:20,232 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/e0d03acaa884442daaa8df2655c9b0f9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/e0d03acaa884442daaa8df2655c9b0f9 2023-04-28 08:54:20,239 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/e0d03acaa884442daaa8df2655c9b0f9, entries=150, sequenceid=149, filesize=11.8 K 2023-04-28 08:54:20,240 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 1677ms, sequenceid=149, compaction requested=true 2023-04-28 08:54:20,240 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:20,240 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:20,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:20,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:20,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:20,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:20,242 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,242 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,242 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,242 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:20,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2634bfd[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:20,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:20,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,245 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,245 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:20,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:20,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:20,246 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,246 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:20,247 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:20,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:20,250 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:20,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:20,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:20,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:20,250 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:20,252 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,253 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:20,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,253 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:20,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:20,260 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:20,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:20,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:20,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:20,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,261 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,261 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,261 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,261 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,261 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:20,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:20,261 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:20,361 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:20,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:20,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:20,365 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:20,365 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:20,365 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:20,365 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:20,365 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,365 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,366 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:20,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:20,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:20,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:20,368 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:20,368 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:20,368 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:20,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:20,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:20,369 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:20,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:20,784 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304280860c3837bb44533b57096d388369a12 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280860c3837bb44533b57096d388369a12 2023-04-28 08:54:20,790 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/e8e287da42984b68b634aeae7828671b 2023-04-28 08:54:20,802 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/8528951a56b243089e182d96ce6d0763 2023-04-28 08:54:20,816 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/690265fbf859425eb7df55ed17d23534 2023-04-28 08:54:20,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:20,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:20,820 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/e8e287da42984b68b634aeae7828671b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e8e287da42984b68b634aeae7828671b 2023-04-28 08:54:20,826 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e8e287da42984b68b634aeae7828671b, entries=150, sequenceid=169, filesize=25.3 K 2023-04-28 08:54:20,827 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/8528951a56b243089e182d96ce6d0763 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8528951a56b243089e182d96ce6d0763 2023-04-28 08:54:20,832 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8528951a56b243089e182d96ce6d0763, entries=150, sequenceid=169, filesize=11.8 K 2023-04-28 08:54:20,833 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/690265fbf859425eb7df55ed17d23534 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/690265fbf859425eb7df55ed17d23534 2023-04-28 08:54:20,838 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/690265fbf859425eb7df55ed17d23534, entries=150, sequenceid=169, filesize=11.8 K 2023-04-28 08:54:20,839 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 470ms, sequenceid=169, compaction requested=true 2023-04-28 08:54:20,839 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:20,839 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:20,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:20,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:20,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:20,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:20,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:20,841 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:54:20,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:20,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:20,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:20,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:20,841 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:20,841 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:20,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34cc5bce[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:20,845 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:20,845 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,847 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,848 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:20,848 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:20,848 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:20,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:20,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,849 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:20,849 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,849 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:20,853 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:20,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:20,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:20,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:20,853 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:20,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:20,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,857 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,860 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:20,860 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:20,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:20,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:20,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:20,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:20,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:20,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:20,861 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:20,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:20,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,861 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:20,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672120864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672120865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672120866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672120867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672120927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,961 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:20,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:20,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:20,965 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:20,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:20,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:20,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:20,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:20,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:20,969 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,969 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672120968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:20,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:20,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:20,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:20,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:20,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:20,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:20,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:20,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:20,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:20,972 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:20,972 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:20,972 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:20,972 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:20,972 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,972 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:20,972 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:20,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,973 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,974 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,974 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672120971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672120972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:20,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dba0a12[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:20,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:20,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:20,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672120975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,978 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:20,978 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:20,978 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:20,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:20,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:20,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:20,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:20,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:20,983 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:20,983 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:20,983 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:20,983 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:20,983 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:20,986 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,986 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:20,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,986 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:20,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:20,987 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:20,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:20,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:20,990 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:20,990 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:20,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:20,990 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:20,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:20,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:20,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:20,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:20,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:20,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:20,991 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:20,991 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:20,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,091 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,095 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,095 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,095 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,095 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,095 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,095 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,097 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,101 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,101 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,101 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,101 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bfe2440[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,103 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,103 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,105 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,105 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:21,105 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:21,105 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,105 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:21,105 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:21,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:21,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:21,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:21,110 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:21,110 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:21,110 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:21,110 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:21,110 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:21,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,112 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,115 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,117 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:21,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:21,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,117 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:21,117 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:21,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672121170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672121176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672121177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672121178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,218 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,221 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,222 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,222 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,230 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,231 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,231 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cd8eecf[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,232 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,232 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,233 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:21,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:21,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:21,234 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,234 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:21,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,235 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,235 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,235 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:21,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:21,237 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:21,238 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:21,238 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:21,238 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:21,238 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:21,242 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,242 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,242 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:21,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,246 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,246 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:21,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:21,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,246 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:21,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,247 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,247 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,247 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,267 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428bda942b8d338422d9d80868618b5cbc9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bda942b8d338422d9d80868618b5cbc9 2023-04-28 08:54:21,347 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,350 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,354 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,354 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,354 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,354 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,356 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d988d52[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,363 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:21,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:21,363 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,363 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:21,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:21,364 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,365 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:21,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:21,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:21,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:21,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:21,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:21,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:21,368 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,368 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,369 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,371 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,375 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:21,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,376 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:21,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,376 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,476 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672121476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,480 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,485 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672121481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601ffc69[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,488 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:21,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:21,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:21,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:21,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:21,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:21,492 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:21,492 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:21,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,492 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:21,492 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:21,492 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:21,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672121482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672121482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,497 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,497 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,497 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,498 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,502 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,502 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,502 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,502 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,502 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:21,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:21,502 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,502 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,503 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,503 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,503 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,503 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:21,503 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:21,603 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,607 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,608 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,608 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,610 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,610 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,610 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,610 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,611 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,612 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,612 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad9080e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,614 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,614 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,618 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,618 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,618 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:21,618 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:21,618 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:21,618 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:21,620 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:21,620 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,620 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,620 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:21,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:21,622 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:21,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:21,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:21,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:21,622 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:21,624 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,624 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,624 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:21,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:21,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,630 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,630 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:21,676 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/448d857b27ec499989e935384a6ef19a 2023-04-28 08:54:21,730 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,734 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,736 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,737 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,737 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,738 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,739 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,739 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,739 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,740 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@616c3f0c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,742 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,742 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,743 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:21,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:21,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:21,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:21,745 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,745 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:21,745 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,745 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:21,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:21,747 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:21,747 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:21,747 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:21,747 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:21,747 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:21,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:21,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,759 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,759 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,759 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,760 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:21,859 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,863 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,864 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,865 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,866 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,866 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,867 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,867 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,867 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,867 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,867 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,867 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,867 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,867 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,868 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,868 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,869 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,869 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2857dc6e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,870 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,870 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,873 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,873 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:21,873 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:21,873 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,873 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:21,873 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:21,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:21,878 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:21,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:21,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:21,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:21,878 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:21,879 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,879 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,879 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,881 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:21,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:21,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:21,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,886 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:21,886 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:21,886 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:21,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672121980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,986 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:21,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:21,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:21,989 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:21,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:21,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:21,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672121988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,992 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:21,992 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:21,992 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,992 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:21,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:21,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:21,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:21,994 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,994 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:21,994 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:21,994 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:21,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:21,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,995 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:21,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672121995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:21,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672121996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,997 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:21,997 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:21,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:21,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:21,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@373481b9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:21,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:22,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:22,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:22,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:22,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:22,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,001 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:22,004 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:22,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:22,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:22,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:22,004 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:22,009 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:22,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:22,014 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:22,014 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:22,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:22,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/e42a4c5bd4794580866da6d43b5cf2cb 2023-04-28 08:54:22,114 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:22,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:22,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:22,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:22,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:22,118 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:22,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:22,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:22,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,119 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:22,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:22,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:22,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:22,121 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,121 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:22,121 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:22,121 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:22,121 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:22,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,123 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:22,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f6629[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:22,124 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:22,124 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:22,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:22,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:22,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:22,128 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:22,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:22,131 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:22,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:22,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:22,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:22,131 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:22,132 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,132 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,132 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,133 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:22,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:22,137 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,137 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,237 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:22,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:22,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:22,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:22,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:22,241 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:22,241 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,241 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:22,241 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:22,242 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,242 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,244 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:22,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:22,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:22,245 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:22,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:22,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:22,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:22,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:22,248 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,248 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,248 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,248 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:22,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58bbdd9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:22,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:22,250 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,251 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:22,251 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,251 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:22,251 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:22,251 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:22,252 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,252 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,252 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,252 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:22,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:22,255 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:22,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:22,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:22,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:22,255 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:22,258 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,258 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,258 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,258 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,260 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,262 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:22,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:22,262 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:22,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,262 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:22,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,263 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,263 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,263 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,362 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:22,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:22,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:22,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:22,366 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:22,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:22,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:22,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,368 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,368 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:22,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:22,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:22,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:22,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:22,369 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,370 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:22,370 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:22,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:22,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1329b5aa[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:22,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:22,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,375 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:22,375 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:22,375 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:22,375 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:22,377 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:22,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:22,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:22,379 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:22,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:22,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:22,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:22,379 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:22,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:22,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:22,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:22,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,388 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,389 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:22,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,389 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:22,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,489 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:22,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:22,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:22,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:22,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:22,493 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:22,493 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,493 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:22,493 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:22,493 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,493 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,494 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:22,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:22,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:22,496 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,496 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:22,496 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:22,496 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:22,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:22,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,498 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/ceb204d0256647ebb597a16e68f6b8de 2023-04-28 08:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83d73f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:22,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:22,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,501 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,501 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:22,501 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:22,501 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:22,502 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,502 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:22,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/448d857b27ec499989e935384a6ef19a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/448d857b27ec499989e935384a6ef19a 2023-04-28 08:54:22,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/448d857b27ec499989e935384a6ef19a, entries=450, sequenceid=182, filesize=66.3 K 2023-04-28 08:54:22,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:22,509 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/e42a4c5bd4794580866da6d43b5cf2cb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/e42a4c5bd4794580866da6d43b5cf2cb 2023-04-28 08:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:22,516 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:22,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:22,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:22,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:22,516 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:22,532 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,532 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,532 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:22,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:22,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,533 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:22,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/e42a4c5bd4794580866da6d43b5cf2cb, entries=150, sequenceid=182, filesize=11.8 K 2023-04-28 08:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/ceb204d0256647ebb597a16e68f6b8de as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ceb204d0256647ebb597a16e68f6b8de 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:22,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:22,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,538 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:22,538 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:22,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:22,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ceb204d0256647ebb597a16e68f6b8de, entries=150, sequenceid=182, filesize=11.8 K 2023-04-28 08:54:22,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 1700ms, sequenceid=182, compaction requested=true 2023-04-28 08:54:22,541 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:22,541 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:22,541 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:22,541 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-28 08:54:22,541 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:22,541 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-28 08:54:22,544 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 281305 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-28 08:54:22,545 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107677 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-28 08:54:22,545 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/A is initiating minor compaction (all files) 2023-04-28 08:54:22,545 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/B is initiating minor compaction (all files) 2023-04-28 08:54:22,545 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/A in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,545 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/B in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,545 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8ac31ee9736248519f6276fc744a515c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/a85a15304ce54856871b85bd262ffdd1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8baca76ca8e74d2e9c185c6497e71b66, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/86898ed11af2498eaee798968ec50cf7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/becfd2c8fa344c8abf323e23af064750, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/35e614ab4f0b414181ac465543dafd5f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/7b71690ee46746e18f15e768500bc3f2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e8e287da42984b68b634aeae7828671b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/448d857b27ec499989e935384a6ef19a] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=274.7 K 2023-04-28 08:54:22,545 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2597f746b83c4357af9062ee7025013d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f4253a3147a346bca87868e82ef2c0e0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0f10c83d7077409783a99b6cd165256e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f6ba6f2ebd8846c7bc37744c5f9e72c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8221568c8b284388a023ee87a0982e6c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/43e060380e664c1590aee2654a9d1c53, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/510ec16b67824d459ff144f1d02e63fa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8528951a56b243089e182d96ce6d0763, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/e42a4c5bd4794580866da6d43b5cf2cb] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=105.2 K 2023-04-28 08:54:22,545 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8ac31ee9736248519f6276fc744a515c, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672053019 2023-04-28 08:54:22,545 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2597f746b83c4357af9062ee7025013d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672053019 2023-04-28 08:54:22,546 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a85a15304ce54856871b85bd262ffdd1, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682672053365 2023-04-28 08:54:22,546 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting f4253a3147a346bca87868e82ef2c0e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682672053378 2023-04-28 08:54:22,546 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8baca76ca8e74d2e9c185c6497e71b66, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682672053711 2023-04-28 08:54:22,546 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0f10c83d7077409783a99b6cd165256e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682672053711 2023-04-28 08:54:22,547 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 86898ed11af2498eaee798968ec50cf7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682672054887 2023-04-28 08:54:22,547 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting f6ba6f2ebd8846c7bc37744c5f9e72c2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682672054887 2023-04-28 08:54:22,547 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting becfd2c8fa344c8abf323e23af064750, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682672055512 2023-04-28 08:54:22,547 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8221568c8b284388a023ee87a0982e6c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682672055512 2023-04-28 08:54:22,548 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 35e614ab4f0b414181ac465543dafd5f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1682672056674 2023-04-28 08:54:22,548 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 43e060380e664c1590aee2654a9d1c53, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1682672056674 2023-04-28 08:54:22,548 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7b71690ee46746e18f15e768500bc3f2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672057578 2023-04-28 08:54:22,548 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 510ec16b67824d459ff144f1d02e63fa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672057578 2023-04-28 08:54:22,548 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e8e287da42984b68b634aeae7828671b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682672058782 2023-04-28 08:54:22,548 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8528951a56b243089e182d96ce6d0763, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682672058782 2023-04-28 08:54:22,549 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 448d857b27ec499989e935384a6ef19a, keycount=450, bloomtype=ROW, size=66.3 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682672060815 2023-04-28 08:54:22,549 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e42a4c5bd4794580866da6d43b5cf2cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682672060829 2023-04-28 08:54:22,568 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#B#compaction#156 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:22,574 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#A#compaction#155 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:22,638 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:22,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:22,642 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:22,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:22,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:22,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,643 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:22,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:22,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:22,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:22,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:22,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:22,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:22,645 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:22,646 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:22,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:22,647 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:22,647 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:22,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:22,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:22,989 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/85adb57e19134a8fa20a596a233623f3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/85adb57e19134a8fa20a596a233623f3 2023-04-28 08:54:22,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:22,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672122989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672122995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:22,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672122996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:22,997 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/B of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 85adb57e19134a8fa20a596a233623f3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:22,997 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:22,997 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/B, priority=7, startTime=1682672062541; duration=0sec 2023-04-28 08:54:22,997 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:22,997 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-28 08:54:23,001 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 107677 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-28 08:54:23,001 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/C is initiating minor compaction (all files) 2023-04-28 08:54:23,001 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/C in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:23,001 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/64a461af211c4f9ba28a54eed29b4f1e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/744edc0fe207438a84a02f3ad6b4949a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b6e710ae44984fe2b30119e4ab5aef41, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/beb7e451c9f9459a97280486cf70b46f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5a55a6ad54614f80aac771b221fec013, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/07c1dcb1570e4d2b989a113022f716b1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/e0d03acaa884442daaa8df2655c9b0f9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/690265fbf859425eb7df55ed17d23534, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ceb204d0256647ebb597a16e68f6b8de] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=105.2 K 2023-04-28 08:54:23,002 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 64a461af211c4f9ba28a54eed29b4f1e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672053019 2023-04-28 08:54:23,002 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 744edc0fe207438a84a02f3ad6b4949a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1682672053378 2023-04-28 08:54:23,003 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b6e710ae44984fe2b30119e4ab5aef41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1682672053711 2023-04-28 08:54:23,003 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting beb7e451c9f9459a97280486cf70b46f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1682672054887 2023-04-28 08:54:23,004 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5a55a6ad54614f80aac771b221fec013, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1682672055512 2023-04-28 08:54:23,004 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 07c1dcb1570e4d2b989a113022f716b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1682672056674 2023-04-28 08:54:23,004 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e0d03acaa884442daaa8df2655c9b0f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672057578 2023-04-28 08:54:23,005 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 690265fbf859425eb7df55ed17d23534, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1682672058782 2023-04-28 08:54:23,005 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ceb204d0256647ebb597a16e68f6b8de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682672060829 2023-04-28 08:54:23,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672123003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672123007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,019 WARN [DataStreamer for file /user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428eb1cc8b749d24db7bb27fa785368343a] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-28 08:54:23,041 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#C#compaction#158 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:23,048 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/6976b02404fc40a58decf4365d47bc8b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/6976b02404fc40a58decf4365d47bc8b 2023-04-28 08:54:23,056 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/A of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 6976b02404fc40a58decf4365d47bc8b(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:23,056 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:23,056 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/A, priority=7, startTime=1682672062541; duration=0sec 2023-04-28 08:54:23,056 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:23,060 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/ac317cc72c58435dbbdbb39f7ffc7f79 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ac317cc72c58435dbbdbb39f7ffc7f79 2023-04-28 08:54:23,064 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304287b1675f8eecc4bcfb206d9501cc50d77 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287b1675f8eecc4bcfb206d9501cc50d77 2023-04-28 08:54:23,068 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/C of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into ac317cc72c58435dbbdbb39f7ffc7f79(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:23,068 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:23,068 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/C, priority=7, startTime=1682672062541; duration=0sec 2023-04-28 08:54:23,068 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:23,077 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/2c77a49b65b34f3885013811a4e12b09 2023-04-28 08:54:23,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672123097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672123097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,106 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/d7636e69e5874614b3a076665d5fde7e 2023-04-28 08:54:23,118 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/c1be8154521b441fae68a2f2f3569d97 2023-04-28 08:54:23,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/2c77a49b65b34f3885013811a4e12b09 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2c77a49b65b34f3885013811a4e12b09 2023-04-28 08:54:23,128 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2c77a49b65b34f3885013811a4e12b09, entries=150, sequenceid=201, filesize=25.3 K 2023-04-28 08:54:23,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/d7636e69e5874614b3a076665d5fde7e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d7636e69e5874614b3a076665d5fde7e 2023-04-28 08:54:23,135 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d7636e69e5874614b3a076665d5fde7e, entries=150, sequenceid=201, filesize=11.8 K 2023-04-28 08:54:23,136 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/c1be8154521b441fae68a2f2f3569d97 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/c1be8154521b441fae68a2f2f3569d97 2023-04-28 08:54:23,140 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/c1be8154521b441fae68a2f2f3569d97, entries=150, sequenceid=201, filesize=11.8 K 2023-04-28 08:54:23,141 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 494ms, sequenceid=201, compaction requested=false 2023-04-28 08:54:23,141 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:23,141 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:23,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:23,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:23,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:23,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:23,143 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,143 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:23,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ed3d728[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:23,145 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:23,145 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:23,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,148 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:23,148 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:23,148 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:23,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,149 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:23,149 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,149 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:23,152 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:23,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:23,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:23,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:23,152 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:23,155 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,155 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,155 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:23,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,156 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,163 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,163 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:23,163 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:23,163 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:23,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:23,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:23,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:23,163 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,163 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,164 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:23,164 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:23,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:23,264 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:23,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:23,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:23,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:23,279 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:23,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:23,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:23,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:23,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,280 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:23,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:23,281 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,281 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:23,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:23,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:23,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:23,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:23,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:23,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:23,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:23,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:23,282 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:54:23,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:23,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:23,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:23,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:23,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:23,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:23,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:23,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:23,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672123327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672123330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672123431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672123434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672123634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:23,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672123638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,711 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e8e08bfb792b4571bf21e909412580af to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e8e08bfb792b4571bf21e909412580af 2023-04-28 08:54:23,750 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=217, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/dd96fd21c87d4508a06e7212cece15db 2023-04-28 08:54:23,774 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/05a2c71cfe9b4fa0a1897486d9efe117 2023-04-28 08:54:23,801 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/2895b7c43acd4d6c846b5c53ccba42e6 2023-04-28 08:54:23,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/dd96fd21c87d4508a06e7212cece15db as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/dd96fd21c87d4508a06e7212cece15db 2023-04-28 08:54:23,818 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/dd96fd21c87d4508a06e7212cece15db, entries=150, sequenceid=217, filesize=25.3 K 2023-04-28 08:54:23,819 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/05a2c71cfe9b4fa0a1897486d9efe117 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/05a2c71cfe9b4fa0a1897486d9efe117 2023-04-28 08:54:23,836 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/05a2c71cfe9b4fa0a1897486d9efe117, entries=150, sequenceid=217, filesize=11.8 K 2023-04-28 08:54:23,837 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/2895b7c43acd4d6c846b5c53ccba42e6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2895b7c43acd4d6c846b5c53ccba42e6 2023-04-28 08:54:23,841 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2895b7c43acd4d6c846b5c53ccba42e6, entries=150, sequenceid=217, filesize=11.8 K 2023-04-28 08:54:23,842 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 560ms, sequenceid=217, compaction requested=true 2023-04-28 08:54:23,842 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:23,842 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:23,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:23,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:23,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:23,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:23,846 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,846 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,847 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,847 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266ec85e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:23,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:23,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:23,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:23,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:23,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:23,853 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,853 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:23,853 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:23,853 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:23,856 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:23,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:23,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:23,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:23,856 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:23,858 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,858 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:23,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,858 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,863 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:23,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:23,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:23,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:23,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,870 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:23,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:23,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:23,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:23,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:23,871 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:23,871 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:23,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:23,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:23,940 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:23,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:23,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:23,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:23,941 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:23,941 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:23,971 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:23,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:23,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:23,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:23,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:23,975 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:23,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:23,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:23,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:23,975 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042828a0a102ce534b009ef29b2b155aadc9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042828a0a102ce534b009ef29b2b155aadc9 2023-04-28 08:54:23,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:23,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:23,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:23,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:23,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:23,979 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:23,979 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:23,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:23,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:23,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:23,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:23,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:23,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:23,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:23,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:23,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:23,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,987 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:23,987 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:23,988 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,988 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7834bddc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:23,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:23,990 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:23,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:23,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:23,992 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,992 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:23,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:23,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:23,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:23,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:23,994 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:23,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:23,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:23,998 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:23,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:23,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:23,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:23,998 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:24,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672123995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672123996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,001 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,001 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,001 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,002 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,007 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:24,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,008 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:24,008 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672124101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672124101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,108 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:24,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:24,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:24,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:24,111 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:24,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:24,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:24,112 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,112 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,113 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,114 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:24,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:24,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:24,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:24,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:24,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:24,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:24,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:24,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:24,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:24,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:24,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:24,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:24,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:24,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:24,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,118 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,118 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:24,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6610a7ca[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:24,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:24,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,122 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:24,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:24,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:24,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:24,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:24,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:24,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:24,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:24,127 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:24,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:24,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:24,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:24,127 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:24,129 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,129 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,129 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,136 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,138 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:24,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:24,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,139 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:24,139 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:24,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,239 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:24,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:24,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:24,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:24,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:24,243 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:24,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:24,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:24,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,244 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:24,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:24,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:24,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:24,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:24,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:24,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:24,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:24,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:24,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:24,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:24,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:24,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:24,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:24,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,250 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,250 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,250 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:24,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a56b7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:24,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:24,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,253 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,253 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:24,253 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:24,253 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:24,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:24,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:24,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:24,259 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:24,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:24,263 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:24,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:24,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:24,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:24,263 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:24,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,266 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,268 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,270 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,270 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,270 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,270 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,270 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,270 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:24,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:24,273 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,273 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:24,274 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:24,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672124305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672124307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,374 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:24,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:24,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:24,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:24,377 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:24,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:24,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:24,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,379 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:24,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:24,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:24,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:24,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:24,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:24,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:24,381 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,381 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:24,382 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:24,382 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:24,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:24,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:24,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:24,387 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:24,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56665202[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:24,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:24,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:24,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:24,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:24,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:24,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:24,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,395 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/4d53b65ea632447eac28d1dd5f7c8623 2023-04-28 08:54:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:24,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:24,395 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:24,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:24,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:24,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:24,395 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:24,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,401 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,405 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,406 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,406 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:24,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:24,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:24,407 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:24,407 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:24,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/058a9ce747c84e4c86c199c7fd05ccff 2023-04-28 08:54:24,507 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:24,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:24,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:24,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:24,514 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:24,514 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,514 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:24,514 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:24,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,516 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:24,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:24,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:24,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:24,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:24,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:24,518 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,518 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:24,518 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:24,518 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:24,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:24,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:24,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:24,520 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,520 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,520 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,520 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b61df4f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:24,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:24,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,523 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:24,523 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:24,523 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:24,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:24,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:24,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,526 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:24,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:24,528 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:24,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:24,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:24,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:24,528 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:24,530 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,530 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,530 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,532 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:24,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:24,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:24,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,537 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:24,537 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:24,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672124609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:24,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672124611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,637 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:24,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:24,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:24,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:24,640 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:24,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:24,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:24,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:24,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:24,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:24,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:24,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:24,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:24,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:24,644 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,644 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:24,644 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:24,644 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:24,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:24,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,646 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,647 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:24,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68ac171e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:24,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:24,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:24,650 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,650 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:24,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:24,650 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:24,651 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:24,651 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,651 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,651 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:24,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:24,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:24,654 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:24,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:24,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:24,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:24,654 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:24,656 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,656 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,656 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,659 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,670 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,670 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,670 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,670 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:24,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:24,670 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,671 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:24,671 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,771 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:24,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:24,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:24,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:24,775 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:24,775 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,775 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:24,775 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:24,775 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,775 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:24,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:24,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:24,778 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:24,778 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,778 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:24,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:24,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:24,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:24,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:24,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:24,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:24,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:24,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:24,780 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,780 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cd7db52[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:24,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:24,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,787 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:24,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:24,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:24,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:24,789 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:24,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:24,792 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:24,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:24,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:24,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:24,792 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:24,793 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,793 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:24,793 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:24,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:24,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,798 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,798 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:24,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,798 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:24,798 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,798 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:24,798 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:24,798 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:24,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:24,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:24,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/6ab5332f61c74337a9bf1dc02fe35a3b 2023-04-28 08:54:24,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/4d53b65ea632447eac28d1dd5f7c8623 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/4d53b65ea632447eac28d1dd5f7c8623 2023-04-28 08:54:24,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/4d53b65ea632447eac28d1dd5f7c8623, entries=200, sequenceid=237, filesize=32.1 K 2023-04-28 08:54:24,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/058a9ce747c84e4c86c199c7fd05ccff as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/058a9ce747c84e4c86c199c7fd05ccff 2023-04-28 08:54:24,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/058a9ce747c84e4c86c199c7fd05ccff, entries=150, sequenceid=237, filesize=11.8 K 2023-04-28 08:54:24,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/6ab5332f61c74337a9bf1dc02fe35a3b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/6ab5332f61c74337a9bf1dc02fe35a3b 2023-04-28 08:54:24,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/6ab5332f61c74337a9bf1dc02fe35a3b, entries=150, sequenceid=237, filesize=11.8 K 2023-04-28 08:54:24,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 908ms, sequenceid=237, compaction requested=true 2023-04-28 08:54:24,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:24,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:24,848 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:24,848 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:24,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:24,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:24,852 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:24,852 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:24,852 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/B is initiating minor compaction (all files) 2023-04-28 08:54:24,852 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/A is initiating minor compaction (all files) 2023-04-28 08:54:24,852 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/B in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,852 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/85adb57e19134a8fa20a596a233623f3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d7636e69e5874614b3a076665d5fde7e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/05a2c71cfe9b4fa0a1897486d9efe117, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/058a9ce747c84e4c86c199c7fd05ccff] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=47.4 K 2023-04-28 08:54:24,852 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/A in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,853 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/6976b02404fc40a58decf4365d47bc8b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2c77a49b65b34f3885013811a4e12b09, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/dd96fd21c87d4508a06e7212cece15db, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/4d53b65ea632447eac28d1dd5f7c8623] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=108.4 K 2023-04-28 08:54:24,853 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 6976b02404fc40a58decf4365d47bc8b, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682672060829 2023-04-28 08:54:24,853 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 85adb57e19134a8fa20a596a233623f3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682672060829 2023-04-28 08:54:24,855 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 2c77a49b65b34f3885013811a4e12b09, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682672060856 2023-04-28 08:54:24,855 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d7636e69e5874614b3a076665d5fde7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682672060856 2023-04-28 08:54:24,855 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 05a2c71cfe9b4fa0a1897486d9efe117, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682672062962 2023-04-28 08:54:24,856 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting dd96fd21c87d4508a06e7212cece15db, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682672062962 2023-04-28 08:54:24,856 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 058a9ce747c84e4c86c199c7fd05ccff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1682672063325 2023-04-28 08:54:24,856 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4d53b65ea632447eac28d1dd5f7c8623, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1682672063324 2023-04-28 08:54:24,868 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#B#compaction#167 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:24,870 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#A#compaction#168 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:24,888 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/a7d260e2f55941ea91acb0152b79ff78 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7d260e2f55941ea91acb0152b79ff78 2023-04-28 08:54:24,895 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/B of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into a7d260e2f55941ea91acb0152b79ff78(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:24,895 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:24,895 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/B, priority=12, startTime=1682672064848; duration=0sec 2023-04-28 08:54:24,895 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:24,896 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:24,898 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:24,898 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/C is initiating minor compaction (all files) 2023-04-28 08:54:24,898 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/C in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,898 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ac317cc72c58435dbbdbb39f7ffc7f79, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/c1be8154521b441fae68a2f2f3569d97, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2895b7c43acd4d6c846b5c53ccba42e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/6ab5332f61c74337a9bf1dc02fe35a3b] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=47.4 K 2023-04-28 08:54:24,898 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:24,899 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ac317cc72c58435dbbdbb39f7ffc7f79, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1682672060829 2023-04-28 08:54:24,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:24,899 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c1be8154521b441fae68a2f2f3569d97, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1682672060856 2023-04-28 08:54:24,900 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2895b7c43acd4d6c846b5c53ccba42e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1682672062962 2023-04-28 08:54:24,900 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6ab5332f61c74337a9bf1dc02fe35a3b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1682672063325 2023-04-28 08:54:24,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:24,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:24,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:24,907 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:24,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:24,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:24,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,909 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:24,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:24,909 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:24,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:24,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:24,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:24,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:24,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:24,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:24,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:24,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:24,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:24,916 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#C#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:24,917 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:24,917 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:24,922 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:24,922 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:24,922 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:24,922 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:24,922 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:24,922 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:24,922 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:24,923 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:24,923 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:24,951 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/8f733deab925420fae31a7c5fe3686bf as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f733deab925420fae31a7c5fe3686bf 2023-04-28 08:54:24,956 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/C of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 8f733deab925420fae31a7c5fe3686bf(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:24,956 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:24,956 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/C, priority=12, startTime=1682672064848; duration=0sec 2023-04-28 08:54:24,956 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:25,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:25,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:25,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672125040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672125043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672125043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672125113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58010 deadline: 1682672125116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672125144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672125148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672125148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672125347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672125353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672125353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,373 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042881918221f6c940dcb0761d8a54058f58 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042881918221f6c940dcb0761d8a54058f58 2023-04-28 08:54:25,378 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/881db1edd9354f57a3039ab689f172cd 2023-04-28 08:54:25,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672125650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672125655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:25,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672125658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,713 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/fb7df5c1b64d4acf815ebea7b2dbdb1d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fb7df5c1b64d4acf815ebea7b2dbdb1d 2023-04-28 08:54:25,719 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/A of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into fb7df5c1b64d4acf815ebea7b2dbdb1d(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:25,719 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:25,719 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/A, priority=12, startTime=1682672064848; duration=0sec 2023-04-28 08:54:25,719 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:25,797 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/7ad65abc5b6c42bcae61b792a4385592 2023-04-28 08:54:25,808 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/ce9b1df2fd254fab8917e172eb137efa 2023-04-28 08:54:25,813 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/881db1edd9354f57a3039ab689f172cd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/881db1edd9354f57a3039ab689f172cd 2023-04-28 08:54:25,818 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/881db1edd9354f57a3039ab689f172cd, entries=150, sequenceid=250, filesize=25.3 K 2023-04-28 08:54:25,819 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/7ad65abc5b6c42bcae61b792a4385592 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7ad65abc5b6c42bcae61b792a4385592 2023-04-28 08:54:25,823 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7ad65abc5b6c42bcae61b792a4385592, entries=150, sequenceid=250, filesize=11.8 K 2023-04-28 08:54:25,824 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/ce9b1df2fd254fab8917e172eb137efa as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ce9b1df2fd254fab8917e172eb137efa 2023-04-28 08:54:25,832 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ce9b1df2fd254fab8917e172eb137efa, entries=150, sequenceid=250, filesize=11.8 K 2023-04-28 08:54:25,833 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 911ms, sequenceid=250, compaction requested=false 2023-04-28 08:54:25,833 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:25,833 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:25,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:25,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:25,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:25,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:25,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,835 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:25,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:25,835 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,835 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:25,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58456888[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:25,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:25,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,839 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:25,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:25,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:25,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:25,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:25,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:25,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:25,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:25,844 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:25,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:25,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:25,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:25,844 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:25,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:25,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:25,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:25,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:25,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:25,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:25,853 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:25,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:25,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:25,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:25,853 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:25,853 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:25,953 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:25,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:25,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:25,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:25,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:25,956 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:25,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:25,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:25,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:25,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:25,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:25,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:25,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:25,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:25,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:25,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:25,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:25,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:25,960 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:25,960 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:25,960 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:25,960 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:25,960 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:25,960 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:25,960 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:25,960 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:25,961 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:25,961 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:25,961 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:25,961 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:25,961 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:25,961 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:25,978 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428844686e79adc44d4a1b01fb7a476b5b9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428844686e79adc44d4a1b01fb7a476b5b9 2023-04-28 08:54:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:26,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:26,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58078 deadline: 1682672126135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672126136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672126157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672126159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672126162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672126241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672126241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,387 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/fe0cf144d7e446b0a1656740234eb2dd 2023-04-28 08:54:26,412 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/0da965ac90ce49d18a23f1e9d64a0537 2023-04-28 08:54:26,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672126443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672126443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672126747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:26,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672126747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,831 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/7f38af2092a8493b99530230808ba734 2023-04-28 08:54:26,838 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/fe0cf144d7e446b0a1656740234eb2dd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fe0cf144d7e446b0a1656740234eb2dd 2023-04-28 08:54:26,853 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fe0cf144d7e446b0a1656740234eb2dd, entries=150, sequenceid=272, filesize=25.4 K 2023-04-28 08:54:26,854 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/0da965ac90ce49d18a23f1e9d64a0537 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0da965ac90ce49d18a23f1e9d64a0537 2023-04-28 08:54:26,859 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0da965ac90ce49d18a23f1e9d64a0537, entries=150, sequenceid=272, filesize=11.9 K 2023-04-28 08:54:26,860 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/7f38af2092a8493b99530230808ba734 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7f38af2092a8493b99530230808ba734 2023-04-28 08:54:26,864 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7f38af2092a8493b99530230808ba734, entries=150, sequenceid=272, filesize=11.9 K 2023-04-28 08:54:26,865 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 905ms, sequenceid=272, compaction requested=true 2023-04-28 08:54:26,865 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:26,865 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:26,865 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:26,865 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:26,865 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,865 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:26,865 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:26,867 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:26,868 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,868 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:26,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5b5f68[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:26,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:26,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,872 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,872 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:26,872 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:26,872 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:26,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:26,873 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:26,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:26,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:26,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:26,877 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:26,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:26,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:26,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:26,878 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:26,879 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,879 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:26,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,880 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,884 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:26,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:26,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:26,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,887 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:26,887 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:26,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:26,987 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:26,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:26,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:26,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:26,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:26,991 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:26,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:26,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:26,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:26,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:26,993 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:26,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:26,994 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:26,994 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:26,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:26,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:26,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:26,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:26,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:26,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:26,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:26,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:26,995 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:26,995 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:26,995 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:26,995 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:26,995 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:26,996 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:26,996 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:26,996 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:26,996 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:27,020 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ec882abf1fe14e7e8c42b1cb5e1d2785 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ec882abf1fe14e7e8c42b1cb5e1d2785 2023-04-28 08:54:27,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:27,166 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:27,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672127190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672127191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672127192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672127251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672127254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672127294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672127295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672127295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,429 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=284, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/ab2d81c3dfbc4b0fa5847983b1eddb17 2023-04-28 08:54:27,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672127498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672127498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672127499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58048 deadline: 1682672127802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672127803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:27,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672127804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,844 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/7077447b3c1f4bb1a0a89d0581461691 2023-04-28 08:54:27,858 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/34153f15d44743cbbc0f7fcd818b0de1 2023-04-28 08:54:27,872 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/ab2d81c3dfbc4b0fa5847983b1eddb17 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ab2d81c3dfbc4b0fa5847983b1eddb17 2023-04-28 08:54:27,877 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ab2d81c3dfbc4b0fa5847983b1eddb17, entries=150, sequenceid=284, filesize=25.4 K 2023-04-28 08:54:27,878 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/7077447b3c1f4bb1a0a89d0581461691 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7077447b3c1f4bb1a0a89d0581461691 2023-04-28 08:54:27,882 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7077447b3c1f4bb1a0a89d0581461691, entries=150, sequenceid=284, filesize=11.9 K 2023-04-28 08:54:27,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/34153f15d44743cbbc0f7fcd818b0de1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/34153f15d44743cbbc0f7fcd818b0de1 2023-04-28 08:54:27,890 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/34153f15d44743cbbc0f7fcd818b0de1, entries=150, sequenceid=284, filesize=11.9 K 2023-04-28 08:54:27,890 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 895ms, sequenceid=284, compaction requested=true 2023-04-28 08:54:27,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:27,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:27,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:27,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:27,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:27,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:27,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,892 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:27,893 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,893 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:27,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5547b73a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:27,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:27,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,896 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,896 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:27,896 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:27,896 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:27,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:27,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:27,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:27,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:27,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:27,904 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:27,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:27,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:27,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:27,904 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:27,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:27,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:27,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:27,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:27,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:27,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,909 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:27,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:27,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:27,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:27,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:27,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:27,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:27,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:27,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:27,912 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:27,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:27,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:27,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:27,912 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:27,912 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:28,012 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:28,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:28,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:28,016 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:28,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:28,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:28,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:28,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:28,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:28,018 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:28,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:28,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:28,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:28,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:28,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:28,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:28,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:28,020 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:28,020 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:28,020 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:28,020 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:28,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:28,021 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:28,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:28,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:28,261 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:28,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672128272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672128273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58024 deadline: 1682672128308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58048 deadline: 1682672128308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:58064 deadline: 1682672128311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672128376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672128376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,439 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ee568a4cba524d6797d14c46367e7f3b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ee568a4cba524d6797d14c46367e7f3b 2023-04-28 08:54:28,464 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/450e135882fb40ce83d40f6101e636ef 2023-04-28 08:54:28,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672128583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672128583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,881 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/36ffa173e7c9464c815ed9ae967b2693 2023-04-28 08:54:28,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672128884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:28,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:28,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672128887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,292 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/b08cef4a14d14f528d3a7efb3f82d232 2023-04-28 08:54:29,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/450e135882fb40ce83d40f6101e636ef as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/450e135882fb40ce83d40f6101e636ef 2023-04-28 08:54:29,301 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/450e135882fb40ce83d40f6101e636ef, entries=150, sequenceid=304, filesize=25.4 K 2023-04-28 08:54:29,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/36ffa173e7c9464c815ed9ae967b2693 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/36ffa173e7c9464c815ed9ae967b2693 2023-04-28 08:54:29,307 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/36ffa173e7c9464c815ed9ae967b2693, entries=150, sequenceid=304, filesize=11.9 K 2023-04-28 08:54:29,307 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/b08cef4a14d14f528d3a7efb3f82d232 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b08cef4a14d14f528d3a7efb3f82d232 2023-04-28 08:54:29,311 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b08cef4a14d14f528d3a7efb3f82d232, entries=150, sequenceid=304, filesize=11.9 K 2023-04-28 08:54:29,312 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 1291ms, sequenceid=304, compaction requested=true 2023-04-28 08:54:29,312 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:29,312 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,312 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:29,312 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:29,312 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,312 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:29,312 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:29,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:54:29,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:29,315 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:29,315 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:29,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:29,315 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,315 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:29,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:29,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c8eb8b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:29,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:29,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,318 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:29,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:29,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:29,319 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,319 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:29,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:29,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:29,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:29,325 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:29,325 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:29,325 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:29,325 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:29,326 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:29,327 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,327 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,327 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,330 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,332 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,332 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:29,332 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:29,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58048 deadline: 1682672129336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58024 deadline: 1682672129337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58064 deadline: 1682672129341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672129390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672129391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,432 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:29,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:29,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:29,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:29,436 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:29,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:29,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:29,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:29,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:29,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:29,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:29,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:29,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:29,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:29,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:29,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:29,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:29,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:29,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:29,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:29,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24bac8d2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:29,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:29,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58048 deadline: 1682672129442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58024 deadline: 1682672129442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,445 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:29,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:29,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:29,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:29,447 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:29,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58064 deadline: 1682672129447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:29,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:29,450 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:29,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:29,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:29,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:29,450 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:29,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:29,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:29,461 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:29,461 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:29,461 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,561 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:29,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:29,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:29,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:29,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:29,565 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:29,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:29,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:29,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,567 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:29,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:29,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:29,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:29,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:29,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:29,569 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,569 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:29,569 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:29,569 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:29,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:29,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:29,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:29,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:29,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231d7332[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:29,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:29,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:29,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:29,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:29,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:29,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,576 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,577 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,577 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:29,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:29,579 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:29,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:29,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:29,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:29,579 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:29,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,582 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,583 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:29,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:29,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,588 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:29,589 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:29,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58048 deadline: 1682672129647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58024 deadline: 1682672129647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58064 deadline: 1682672129651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,689 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:29,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:29,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:29,692 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:29,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:29,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:29,693 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,693 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,695 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,695 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,695 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,695 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:29,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:29,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:29,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:29,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:29,697 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,697 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:29,697 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:29,697 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:29,697 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:29,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,699 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,699 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,699 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:29,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b0bee0c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:29,700 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:29,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,701 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,702 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,702 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:29,702 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:29,702 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:29,702 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:29,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,703 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:29,703 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,703 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:29,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:29,705 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:29,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:29,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:29,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:29,706 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:29,708 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,708 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,708 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,717 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:29,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,721 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,721 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,721 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:29,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,730 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304283a7909343a954437a3e4c2f6e4b687bc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283a7909343a954437a3e4c2f6e4b687bc 2023-04-28 08:54:29,821 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:29,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:29,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:29,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:29,825 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:29,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:29,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:29,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,826 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,826 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,826 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,826 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:29,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:29,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:29,827 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:29,828 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,828 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:29,828 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:29,828 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:29,828 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:29,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,829 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,830 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,830 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:29,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8fd055[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:29,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:29,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,833 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:29,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:29,833 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:29,833 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,833 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:29,834 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,834 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:29,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:29,836 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:29,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:29,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:29,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:29,837 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:29,838 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,838 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,838 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,839 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:29,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:29,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:29,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,851 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:29,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,851 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:29,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,951 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:29,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:29,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:29,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:29,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:29,954 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:29,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:29,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:29,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58024 deadline: 1682672129952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58048 deadline: 1682672129953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:29,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58064 deadline: 1682672129954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:29,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:29,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:29,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:29,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:29,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:29,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:29,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:29,961 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,961 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,961 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,961 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:29,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:29,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29816fae[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:29,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:29,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,963 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:29,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:29,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:29,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:29,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:29,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:29,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:29,970 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:29,970 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:29,970 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:29,970 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:29,970 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:29,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:29,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:29,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,973 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:29,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:29,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:29,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:29,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:29,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:29,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:29,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:29,981 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:29,981 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:30,081 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:30,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:30,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:30,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:30,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:30,084 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:30,084 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:30,084 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:30,084 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:30,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:30,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:30,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:30,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:30,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:30,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:30,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:30,087 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:30,087 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:30,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:30,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:30,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:30,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:30,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:30,090 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,090 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:30,091 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:30,091 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:30,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:30,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cd0d4f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:30,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,093 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:30,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:30,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:30,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:30,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:30,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:30,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:30,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:30,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:30,097 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:30,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:30,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:30,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:30,097 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:30,098 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,098 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:30,099 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:30,099 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:30,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:30,100 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,103 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,103 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:30,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:30,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:30,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:30,103 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:30,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:30,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:30,104 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:30,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,104 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:30,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:30,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:30,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:30,136 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/188306c18950497a87a2f5a8016e69c2 2023-04-28 08:54:30,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/54e07c1ec69347a686b48ef674a75619 2023-04-28 08:54:30,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/bd1c32a56a0c4711909c47ba90f3649c 2023-04-28 08:54:30,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/188306c18950497a87a2f5a8016e69c2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/188306c18950497a87a2f5a8016e69c2 2023-04-28 08:54:30,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/188306c18950497a87a2f5a8016e69c2, entries=200, sequenceid=317, filesize=32.3 K 2023-04-28 08:54:30,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/54e07c1ec69347a686b48ef674a75619 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/54e07c1ec69347a686b48ef674a75619 2023-04-28 08:54:30,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/54e07c1ec69347a686b48ef674a75619, entries=150, sequenceid=317, filesize=11.9 K 2023-04-28 08:54:30,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/bd1c32a56a0c4711909c47ba90f3649c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/bd1c32a56a0c4711909c47ba90f3649c 2023-04-28 08:54:30,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/bd1c32a56a0c4711909c47ba90f3649c, entries=150, sequenceid=317, filesize=11.9 K 2023-04-28 08:54:30,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 884ms, sequenceid=317, compaction requested=true 2023-04-28 08:54:30,198 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:30,198 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:30,198 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:54:30,198 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:54:30,198 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:30,198 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:30,200 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163562 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:54:30,200 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73362 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:54:30,200 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/A is initiating minor compaction (all files) 2023-04-28 08:54:30,200 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/B is initiating minor compaction (all files) 2023-04-28 08:54:30,201 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/A in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:30,201 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/B in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:30,201 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fb7df5c1b64d4acf815ebea7b2dbdb1d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/881db1edd9354f57a3039ab689f172cd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fe0cf144d7e446b0a1656740234eb2dd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ab2d81c3dfbc4b0fa5847983b1eddb17, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/450e135882fb40ce83d40f6101e636ef, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/188306c18950497a87a2f5a8016e69c2] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=159.7 K 2023-04-28 08:54:30,201 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7d260e2f55941ea91acb0152b79ff78, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7ad65abc5b6c42bcae61b792a4385592, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0da965ac90ce49d18a23f1e9d64a0537, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7077447b3c1f4bb1a0a89d0581461691, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/36ffa173e7c9464c815ed9ae967b2693, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/54e07c1ec69347a686b48ef674a75619] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=71.6 K 2023-04-28 08:54:30,201 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting fb7df5c1b64d4acf815ebea7b2dbdb1d, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1682672063325 2023-04-28 08:54:30,201 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a7d260e2f55941ea91acb0152b79ff78, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1682672063325 2023-04-28 08:54:30,202 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 881db1edd9354f57a3039ab689f172cd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682672063980 2023-04-28 08:54:30,202 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7ad65abc5b6c42bcae61b792a4385592, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682672063980 2023-04-28 08:54:30,202 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting fe0cf144d7e446b0a1656740234eb2dd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682672065037 2023-04-28 08:54:30,202 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0da965ac90ce49d18a23f1e9d64a0537, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682672065037 2023-04-28 08:54:30,203 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7077447b3c1f4bb1a0a89d0581461691, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682672066131 2023-04-28 08:54:30,203 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ab2d81c3dfbc4b0fa5847983b1eddb17, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682672066131 2023-04-28 08:54:30,203 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 36ffa173e7c9464c815ed9ae967b2693, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672067183 2023-04-28 08:54:30,204 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 450e135882fb40ce83d40f6101e636ef, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672067183 2023-04-28 08:54:30,204 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 188306c18950497a87a2f5a8016e69c2, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682672068262 2023-04-28 08:54:30,204 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:30,205 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 54e07c1ec69347a686b48ef674a75619, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682672068262 2023-04-28 08:54:30,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:30,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:30,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:30,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:30,223 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:30,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:30,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:30,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:30,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,225 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:30,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:30,225 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,225 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,225 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:30,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:30,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:30,241 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#B#compaction#186 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:30,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:30,242 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:30,242 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:30,242 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:30,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:30,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:30,243 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:30,243 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:30,243 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 08:54:30,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:30,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:30,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:30,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:30,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:30,246 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:30,253 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#A#compaction#185 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:30,270 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/5a5cdd0098754619b544c985f2c989c4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/5a5cdd0098754619b544c985f2c989c4 2023-04-28 08:54:30,282 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/B of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 5a5cdd0098754619b544c985f2c989c4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:30,282 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:30,282 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/B, priority=10, startTime=1682672070198; duration=0sec 2023-04-28 08:54:30,282 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:30,282 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:54:30,284 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73362 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:54:30,285 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): e4dc6da0d5be68cdd66fb19ccfc1d9f0/C is initiating minor compaction (all files) 2023-04-28 08:54:30,285 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e4dc6da0d5be68cdd66fb19ccfc1d9f0/C in TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:30,285 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f733deab925420fae31a7c5fe3686bf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ce9b1df2fd254fab8917e172eb137efa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7f38af2092a8493b99530230808ba734, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/34153f15d44743cbbc0f7fcd818b0de1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b08cef4a14d14f528d3a7efb3f82d232, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/bd1c32a56a0c4711909c47ba90f3649c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp, totalSize=71.6 K 2023-04-28 08:54:30,286 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8f733deab925420fae31a7c5fe3686bf, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1682672063325 2023-04-28 08:54:30,286 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ce9b1df2fd254fab8917e172eb137efa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1682672063980 2023-04-28 08:54:30,287 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7f38af2092a8493b99530230808ba734, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1682672065037 2023-04-28 08:54:30,287 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 34153f15d44743cbbc0f7fcd818b0de1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1682672066131 2023-04-28 08:54:30,287 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b08cef4a14d14f528d3a7efb3f82d232, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1682672067183 2023-04-28 08:54:30,288 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bd1c32a56a0c4711909c47ba90f3649c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1682672068262 2023-04-28 08:54:30,323 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): e4dc6da0d5be68cdd66fb19ccfc1d9f0#C#compaction#188 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:30,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:30,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672130414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672130419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58048 deadline: 1682672130457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58024 deadline: 1682672130459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58064 deadline: 1682672130461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672130520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672130520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672130723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:30,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672130724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:30,735 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304286fe18e29f1b14c13846ebd4b257e2d04 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286fe18e29f1b14c13846ebd4b257e2d04 2023-04-28 08:54:30,738 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/0a669f7ea0bf42e792b1acf87d9d7cc5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/0a669f7ea0bf42e792b1acf87d9d7cc5 2023-04-28 08:54:30,743 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/C of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into 0a669f7ea0bf42e792b1acf87d9d7cc5(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:30,743 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:30,743 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/C, priority=10, startTime=1682672070198; duration=0sec 2023-04-28 08:54:30,743 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:31,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:31,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672131028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:31,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672131028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,143 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/d65b977ff6a34cd28e7585a056dc7316 2023-04-28 08:54:31,159 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/82795345252f4d6a833e44223bee8213 2023-04-28 08:54:31,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:31,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58024 deadline: 1682672131464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58048 deadline: 1682672131468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:31,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58064 deadline: 1682672131471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:31,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58078 deadline: 1682672131531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:31,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:58010 deadline: 1682672131532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,541 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/ac54b7c4b8074a11b2ea618596671db0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ac54b7c4b8074a11b2ea618596671db0 2023-04-28 08:54:31,547 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e4dc6da0d5be68cdd66fb19ccfc1d9f0/A of e4dc6da0d5be68cdd66fb19ccfc1d9f0 into ac54b7c4b8074a11b2ea618596671db0(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 08:54:31,547 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:31,548 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0., storeName=e4dc6da0d5be68cdd66fb19ccfc1d9f0/A, priority=10, startTime=1682672070198; duration=1sec 2023-04-28 08:54:31,548 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:31,569 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/5e9ac3108aef4ad9bbf8afd00c7be372 2023-04-28 08:54:31,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/d65b977ff6a34cd28e7585a056dc7316 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/d65b977ff6a34cd28e7585a056dc7316 2023-04-28 08:54:31,578 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/d65b977ff6a34cd28e7585a056dc7316, entries=150, sequenceid=336, filesize=25.4 K 2023-04-28 08:54:31,579 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/82795345252f4d6a833e44223bee8213 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/82795345252f4d6a833e44223bee8213 2023-04-28 08:54:31,584 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/82795345252f4d6a833e44223bee8213, entries=150, sequenceid=336, filesize=11.9 K 2023-04-28 08:54:31,585 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/5e9ac3108aef4ad9bbf8afd00c7be372 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5e9ac3108aef4ad9bbf8afd00c7be372 2023-04-28 08:54:31,589 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5e9ac3108aef4ad9bbf8afd00c7be372, entries=150, sequenceid=336, filesize=11.9 K 2023-04-28 08:54:31,590 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 1347ms, sequenceid=336, compaction requested=false 2023-04-28 08:54:31,590 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:31,590 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:31,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:31,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:31,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:31,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:31,592 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7522e4f4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:31,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:31,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,595 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:31,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:31,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:31,595 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,595 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:31,597 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:31,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:31,600 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:31,600 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:31,600 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:31,600 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:31,600 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:31,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:31,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:31,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:31,602 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,603 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:31,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:31,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:31,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:31,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:31,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:31,605 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:31,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:31,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:31,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,607 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:31,607 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:31,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:31,707 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:31,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:31,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:31,710 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:31,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:31,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:31,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:31,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:31,712 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:31,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:31,712 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:31,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:31,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:31,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:31,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:31,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:31,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:31,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:31,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:31,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:31,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:31,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:31,714 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:31,714 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. started... 2023-04-28 08:54:31,715 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:54:31,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:31,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:31,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:31,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:31,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:31,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:31,728 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428662a66e35c6f4db8af504074c54d46d3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428662a66e35c6f4db8af504074c54d46d3 2023-04-28 08:54:31,789 DEBUG [Thread-932] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c4379c2 to 127.0.0.1:54655 2023-04-28 08:54:31,789 DEBUG [Thread-932] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:31,790 DEBUG [Thread-938] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2efdb02f to 127.0.0.1:54655 2023-04-28 08:54:31,790 DEBUG [Thread-934] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f43b712 to 127.0.0.1:54655 2023-04-28 08:54:31,791 DEBUG [Thread-938] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:31,791 DEBUG [Thread-934] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:31,793 DEBUG [Thread-936] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a3bb940 to 127.0.0.1:54655 2023-04-28 08:54:31,793 DEBUG [Thread-936] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:32,134 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/db029792a3c34648b1fd13ce5c05048a 2023-04-28 08:54:32,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:32,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. as already flushing 2023-04-28 08:54:32,537 DEBUG [Thread-927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6c1c92c6 to 127.0.0.1:54655 2023-04-28 08:54:32,537 DEBUG [Thread-927] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:32,542 DEBUG [Thread-923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1441c9f0 to 127.0.0.1:54655 2023-04-28 08:54:32,542 DEBUG [Thread-923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:32,543 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/d29c6a7cae7143b5a0e9bb6424d27585 2023-04-28 08:54:32,952 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/2df1b850ee00409b85f072f42cc9c76c 2023-04-28 08:54:32,957 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/db029792a3c34648b1fd13ce5c05048a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/db029792a3c34648b1fd13ce5c05048a 2023-04-28 08:54:32,960 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/db029792a3c34648b1fd13ce5c05048a, entries=150, sequenceid=352, filesize=25.4 K 2023-04-28 08:54:32,961 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/d29c6a7cae7143b5a0e9bb6424d27585 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d29c6a7cae7143b5a0e9bb6424d27585 2023-04-28 08:54:32,964 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d29c6a7cae7143b5a0e9bb6424d27585, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 08:54:32,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/2df1b850ee00409b85f072f42cc9c76c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2df1b850ee00409b85f072f42cc9c76c 2023-04-28 08:54:32,968 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2df1b850ee00409b85f072f42cc9c76c, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 08:54:32,969 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=13.42 KB/13740 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 1254ms, sequenceid=352, compaction requested=true 2023-04-28 08:54:32,969 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:32,969 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:32,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:32,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:32,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:32,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:32,970 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,970 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:32,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:32,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@508a9f7d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:32,972 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:32,972 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,973 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:32,973 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:32,973 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,973 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:32,973 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:32,974 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:32,974 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,974 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:32,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:32,974 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:32,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:32,976 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:32,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:32,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:32,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:32,976 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:32,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:32,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:32,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:32,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:32,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:32,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,983 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,983 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:32,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:32,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:32,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:32,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:32,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:32,984 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:32,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:32,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:32,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:32,984 DEBUG [Thread-931] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:32,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,984 DEBUG [Thread-931] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:32,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:32,984 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:33,084 DEBUG [Thread-931] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:33,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:33,476 DEBUG [Thread-921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21281a37 to 127.0.0.1:54655 2023-04-28 08:54:33,476 DEBUG [Thread-921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:33,480 DEBUG [Thread-929] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c472cfd to 127.0.0.1:54655 2023-04-28 08:54:33,480 DEBUG [Thread-929] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:33,492 DEBUG [Thread-925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73e51571 to 127.0.0.1:54655 2023-04-28 08:54:33,492 DEBUG [Thread-925] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:33,492 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 61 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 4599 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 4498 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1942 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5826 rows 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1936 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5804 rows 2023-04-28 08:54:33,493 INFO [Listener at localhost/39717] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 08:54:33,493 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7829b43e to 127.0.0.1:54655 2023-04-28 08:54:33,493 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:33,495 INFO [Listener at localhost/39717] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 08:54:33,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable TestAcidGuarantees 2023-04-28 08:54:33,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:33,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 08:54:33,500 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672073500"}]},"ts":"1682672073500"} 2023-04-28 08:54:33,501 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 08:54:33,502 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 08:54:33,503 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, UNASSIGN}] 2023-04-28 08:54:33,504 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, UNASSIGN 2023-04-28 08:54:33,504 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:33,505 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672073504"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672073504"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672073504"}]},"ts":"1682672073504"} 2023-04-28 08:54:33,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:54:33,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 08:54:33,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:33,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing e4dc6da0d5be68cdd66fb19ccfc1d9f0, disabling compactions & flushes 2023-04-28 08:54:33,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:33,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:33,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. after waiting 0 ms 2023-04-28 08:54:33,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:33,657 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing e4dc6da0d5be68cdd66fb19ccfc1d9f0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 08:54:33,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=A 2023-04-28 08:54:33,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:33,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=B 2023-04-28 08:54:33,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:33,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e4dc6da0d5be68cdd66fb19ccfc1d9f0, store=C 2023-04-28 08:54:33,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:33,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 08:54:34,069 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304283d9ca24103d24e00a009038e2cc4323e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283d9ca24103d24e00a009038e2cc4323e 2023-04-28 08:54:34,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 08:54:34,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/e1ee331fb7384b1f9230dfc024430d6f 2023-04-28 08:54:34,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 08:54:34,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/2436b4a7890d45348e939fb8307f8052 2023-04-28 08:54:35,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/8f50adfb90e342e2b23195282c06dc78 2023-04-28 08:54:35,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/A/e1ee331fb7384b1f9230dfc024430d6f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e1ee331fb7384b1f9230dfc024430d6f 2023-04-28 08:54:35,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e1ee331fb7384b1f9230dfc024430d6f, entries=150, sequenceid=360, filesize=25.4 K 2023-04-28 08:54:35,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/B/2436b4a7890d45348e939fb8307f8052 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2436b4a7890d45348e939fb8307f8052 2023-04-28 08:54:35,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2436b4a7890d45348e939fb8307f8052, entries=150, sequenceid=360, filesize=11.9 K 2023-04-28 08:54:35,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/.tmp/C/8f50adfb90e342e2b23195282c06dc78 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f50adfb90e342e2b23195282c06dc78 2023-04-28 08:54:35,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f50adfb90e342e2b23195282c06dc78, entries=150, sequenceid=360, filesize=11.9 K 2023-04-28 08:54:35,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for e4dc6da0d5be68cdd66fb19ccfc1d9f0 in 1651ms, sequenceid=360, compaction requested=true 2023-04-28 08:54:35,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/08cfbaa077c44b63af2c87497fe1fa51, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ee6df7b3328542a09b44d341390a71eb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8ac31ee9736248519f6276fc744a515c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2e604026032f4252a262e30313b1a062, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/a85a15304ce54856871b85bd262ffdd1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8baca76ca8e74d2e9c185c6497e71b66, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/86898ed11af2498eaee798968ec50cf7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/becfd2c8fa344c8abf323e23af064750, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/35e614ab4f0b414181ac465543dafd5f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/7b71690ee46746e18f15e768500bc3f2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e8e287da42984b68b634aeae7828671b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/448d857b27ec499989e935384a6ef19a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/6976b02404fc40a58decf4365d47bc8b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2c77a49b65b34f3885013811a4e12b09, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/dd96fd21c87d4508a06e7212cece15db, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/4d53b65ea632447eac28d1dd5f7c8623, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fb7df5c1b64d4acf815ebea7b2dbdb1d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/881db1edd9354f57a3039ab689f172cd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fe0cf144d7e446b0a1656740234eb2dd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ab2d81c3dfbc4b0fa5847983b1eddb17, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/450e135882fb40ce83d40f6101e636ef, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/188306c18950497a87a2f5a8016e69c2] to archive 2023-04-28 08:54:35,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:54:35,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/08cfbaa077c44b63af2c87497fe1fa51 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/08cfbaa077c44b63af2c87497fe1fa51 2023-04-28 08:54:35,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ee6df7b3328542a09b44d341390a71eb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ee6df7b3328542a09b44d341390a71eb 2023-04-28 08:54:35,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8ac31ee9736248519f6276fc744a515c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8ac31ee9736248519f6276fc744a515c 2023-04-28 08:54:35,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2e604026032f4252a262e30313b1a062 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2e604026032f4252a262e30313b1a062 2023-04-28 08:54:35,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/a85a15304ce54856871b85bd262ffdd1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/a85a15304ce54856871b85bd262ffdd1 2023-04-28 08:54:35,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8baca76ca8e74d2e9c185c6497e71b66 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/8baca76ca8e74d2e9c185c6497e71b66 2023-04-28 08:54:35,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/86898ed11af2498eaee798968ec50cf7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/86898ed11af2498eaee798968ec50cf7 2023-04-28 08:54:35,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/becfd2c8fa344c8abf323e23af064750 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/becfd2c8fa344c8abf323e23af064750 2023-04-28 08:54:35,340 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/35e614ab4f0b414181ac465543dafd5f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/35e614ab4f0b414181ac465543dafd5f 2023-04-28 08:54:35,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/7b71690ee46746e18f15e768500bc3f2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/7b71690ee46746e18f15e768500bc3f2 2023-04-28 08:54:35,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e8e287da42984b68b634aeae7828671b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e8e287da42984b68b634aeae7828671b 2023-04-28 08:54:35,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/448d857b27ec499989e935384a6ef19a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/448d857b27ec499989e935384a6ef19a 2023-04-28 08:54:35,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/6976b02404fc40a58decf4365d47bc8b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/6976b02404fc40a58decf4365d47bc8b 2023-04-28 08:54:35,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2c77a49b65b34f3885013811a4e12b09 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/2c77a49b65b34f3885013811a4e12b09 2023-04-28 08:54:35,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/dd96fd21c87d4508a06e7212cece15db to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/dd96fd21c87d4508a06e7212cece15db 2023-04-28 08:54:35,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/4d53b65ea632447eac28d1dd5f7c8623 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/4d53b65ea632447eac28d1dd5f7c8623 2023-04-28 08:54:35,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fb7df5c1b64d4acf815ebea7b2dbdb1d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fb7df5c1b64d4acf815ebea7b2dbdb1d 2023-04-28 08:54:35,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/881db1edd9354f57a3039ab689f172cd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/881db1edd9354f57a3039ab689f172cd 2023-04-28 08:54:35,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fe0cf144d7e446b0a1656740234eb2dd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/fe0cf144d7e446b0a1656740234eb2dd 2023-04-28 08:54:35,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ab2d81c3dfbc4b0fa5847983b1eddb17 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ab2d81c3dfbc4b0fa5847983b1eddb17 2023-04-28 08:54:35,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/450e135882fb40ce83d40f6101e636ef to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/450e135882fb40ce83d40f6101e636ef 2023-04-28 08:54:35,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/188306c18950497a87a2f5a8016e69c2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/188306c18950497a87a2f5a8016e69c2 2023-04-28 08:54:35,356 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/3db7ba5b08e74d599ac484cd8d93b8c3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/dc4dc98b534b4f45a776150fcbc0ddfc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2597f746b83c4357af9062ee7025013d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7aa7c37a10343168cda27caa7b621e0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f4253a3147a346bca87868e82ef2c0e0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0f10c83d7077409783a99b6cd165256e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f6ba6f2ebd8846c7bc37744c5f9e72c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8221568c8b284388a023ee87a0982e6c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/43e060380e664c1590aee2654a9d1c53, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/510ec16b67824d459ff144f1d02e63fa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8528951a56b243089e182d96ce6d0763, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/85adb57e19134a8fa20a596a233623f3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/e42a4c5bd4794580866da6d43b5cf2cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d7636e69e5874614b3a076665d5fde7e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/05a2c71cfe9b4fa0a1897486d9efe117, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7d260e2f55941ea91acb0152b79ff78, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/058a9ce747c84e4c86c199c7fd05ccff, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7ad65abc5b6c42bcae61b792a4385592, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0da965ac90ce49d18a23f1e9d64a0537, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7077447b3c1f4bb1a0a89d0581461691, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/36ffa173e7c9464c815ed9ae967b2693, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/54e07c1ec69347a686b48ef674a75619] to archive 2023-04-28 08:54:35,357 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:54:35,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/3db7ba5b08e74d599ac484cd8d93b8c3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/3db7ba5b08e74d599ac484cd8d93b8c3 2023-04-28 08:54:35,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/dc4dc98b534b4f45a776150fcbc0ddfc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/dc4dc98b534b4f45a776150fcbc0ddfc 2023-04-28 08:54:35,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2597f746b83c4357af9062ee7025013d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2597f746b83c4357af9062ee7025013d 2023-04-28 08:54:35,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7aa7c37a10343168cda27caa7b621e0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7aa7c37a10343168cda27caa7b621e0 2023-04-28 08:54:35,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f4253a3147a346bca87868e82ef2c0e0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f4253a3147a346bca87868e82ef2c0e0 2023-04-28 08:54:35,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0f10c83d7077409783a99b6cd165256e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0f10c83d7077409783a99b6cd165256e 2023-04-28 08:54:35,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f6ba6f2ebd8846c7bc37744c5f9e72c2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/f6ba6f2ebd8846c7bc37744c5f9e72c2 2023-04-28 08:54:35,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8221568c8b284388a023ee87a0982e6c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8221568c8b284388a023ee87a0982e6c 2023-04-28 08:54:35,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/43e060380e664c1590aee2654a9d1c53 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/43e060380e664c1590aee2654a9d1c53 2023-04-28 08:54:35,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/510ec16b67824d459ff144f1d02e63fa to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/510ec16b67824d459ff144f1d02e63fa 2023-04-28 08:54:35,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8528951a56b243089e182d96ce6d0763 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/8528951a56b243089e182d96ce6d0763 2023-04-28 08:54:35,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/85adb57e19134a8fa20a596a233623f3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/85adb57e19134a8fa20a596a233623f3 2023-04-28 08:54:35,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/e42a4c5bd4794580866da6d43b5cf2cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/e42a4c5bd4794580866da6d43b5cf2cb 2023-04-28 08:54:35,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d7636e69e5874614b3a076665d5fde7e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d7636e69e5874614b3a076665d5fde7e 2023-04-28 08:54:35,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/05a2c71cfe9b4fa0a1897486d9efe117 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/05a2c71cfe9b4fa0a1897486d9efe117 2023-04-28 08:54:35,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7d260e2f55941ea91acb0152b79ff78 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/a7d260e2f55941ea91acb0152b79ff78 2023-04-28 08:54:35,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/058a9ce747c84e4c86c199c7fd05ccff to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/058a9ce747c84e4c86c199c7fd05ccff 2023-04-28 08:54:35,376 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7ad65abc5b6c42bcae61b792a4385592 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7ad65abc5b6c42bcae61b792a4385592 2023-04-28 08:54:35,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0da965ac90ce49d18a23f1e9d64a0537 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/0da965ac90ce49d18a23f1e9d64a0537 2023-04-28 08:54:35,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7077447b3c1f4bb1a0a89d0581461691 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/7077447b3c1f4bb1a0a89d0581461691 2023-04-28 08:54:35,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/36ffa173e7c9464c815ed9ae967b2693 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/36ffa173e7c9464c815ed9ae967b2693 2023-04-28 08:54:35,381 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/54e07c1ec69347a686b48ef674a75619 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/54e07c1ec69347a686b48ef674a75619 2023-04-28 08:54:35,384 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/99405482cc37403989347210ddc4e24b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7399af4a7c394cd598bd04ae3ae71102, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/64a461af211c4f9ba28a54eed29b4f1e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/4087a608bd724dc08051fb3f1f4f6e0f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/744edc0fe207438a84a02f3ad6b4949a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b6e710ae44984fe2b30119e4ab5aef41, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/beb7e451c9f9459a97280486cf70b46f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5a55a6ad54614f80aac771b221fec013, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/07c1dcb1570e4d2b989a113022f716b1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/e0d03acaa884442daaa8df2655c9b0f9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/690265fbf859425eb7df55ed17d23534, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ac317cc72c58435dbbdbb39f7ffc7f79, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ceb204d0256647ebb597a16e68f6b8de, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/c1be8154521b441fae68a2f2f3569d97, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2895b7c43acd4d6c846b5c53ccba42e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f733deab925420fae31a7c5fe3686bf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/6ab5332f61c74337a9bf1dc02fe35a3b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ce9b1df2fd254fab8917e172eb137efa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7f38af2092a8493b99530230808ba734, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/34153f15d44743cbbc0f7fcd818b0de1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b08cef4a14d14f528d3a7efb3f82d232, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/bd1c32a56a0c4711909c47ba90f3649c] to archive 2023-04-28 08:54:35,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:54:35,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/99405482cc37403989347210ddc4e24b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/99405482cc37403989347210ddc4e24b 2023-04-28 08:54:35,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7399af4a7c394cd598bd04ae3ae71102 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7399af4a7c394cd598bd04ae3ae71102 2023-04-28 08:54:35,390 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/64a461af211c4f9ba28a54eed29b4f1e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/64a461af211c4f9ba28a54eed29b4f1e 2023-04-28 08:54:35,391 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/4087a608bd724dc08051fb3f1f4f6e0f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/4087a608bd724dc08051fb3f1f4f6e0f 2023-04-28 08:54:35,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/744edc0fe207438a84a02f3ad6b4949a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/744edc0fe207438a84a02f3ad6b4949a 2023-04-28 08:54:35,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b6e710ae44984fe2b30119e4ab5aef41 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b6e710ae44984fe2b30119e4ab5aef41 2023-04-28 08:54:35,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/beb7e451c9f9459a97280486cf70b46f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/beb7e451c9f9459a97280486cf70b46f 2023-04-28 08:54:35,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5a55a6ad54614f80aac771b221fec013 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5a55a6ad54614f80aac771b221fec013 2023-04-28 08:54:35,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/07c1dcb1570e4d2b989a113022f716b1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/07c1dcb1570e4d2b989a113022f716b1 2023-04-28 08:54:35,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/e0d03acaa884442daaa8df2655c9b0f9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/e0d03acaa884442daaa8df2655c9b0f9 2023-04-28 08:54:35,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/690265fbf859425eb7df55ed17d23534 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/690265fbf859425eb7df55ed17d23534 2023-04-28 08:54:35,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ac317cc72c58435dbbdbb39f7ffc7f79 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ac317cc72c58435dbbdbb39f7ffc7f79 2023-04-28 08:54:35,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ceb204d0256647ebb597a16e68f6b8de to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ceb204d0256647ebb597a16e68f6b8de 2023-04-28 08:54:35,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/c1be8154521b441fae68a2f2f3569d97 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/c1be8154521b441fae68a2f2f3569d97 2023-04-28 08:54:35,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2895b7c43acd4d6c846b5c53ccba42e6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2895b7c43acd4d6c846b5c53ccba42e6 2023-04-28 08:54:35,405 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f733deab925420fae31a7c5fe3686bf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f733deab925420fae31a7c5fe3686bf 2023-04-28 08:54:35,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/6ab5332f61c74337a9bf1dc02fe35a3b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/6ab5332f61c74337a9bf1dc02fe35a3b 2023-04-28 08:54:35,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ce9b1df2fd254fab8917e172eb137efa to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/ce9b1df2fd254fab8917e172eb137efa 2023-04-28 08:54:35,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7f38af2092a8493b99530230808ba734 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/7f38af2092a8493b99530230808ba734 2023-04-28 08:54:35,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/34153f15d44743cbbc0f7fcd818b0de1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/34153f15d44743cbbc0f7fcd818b0de1 2023-04-28 08:54:35,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b08cef4a14d14f528d3a7efb3f82d232 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/b08cef4a14d14f528d3a7efb3f82d232 2023-04-28 08:54:35,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/bd1c32a56a0c4711909c47ba90f3649c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/bd1c32a56a0c4711909c47ba90f3649c 2023-04-28 08:54:35,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/recovered.edits/363.seqid, newMaxSeqId=363, maxSeqId=4 2023-04-28 08:54:35,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0. 2023-04-28 08:54:35,417 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for e4dc6da0d5be68cdd66fb19ccfc1d9f0: 2023-04-28 08:54:35,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:35,419 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=e4dc6da0d5be68cdd66fb19ccfc1d9f0, regionState=CLOSED 2023-04-28 08:54:35,419 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672075419"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672075419"}]},"ts":"1682672075419"} 2023-04-28 08:54:35,421 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-28 08:54:35,421 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure e4dc6da0d5be68cdd66fb19ccfc1d9f0, server=jenkins-hbase6.apache.org,40909,1682672018517 in 1.9150 sec 2023-04-28 08:54:35,422 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-28 08:54:35,422 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e4dc6da0d5be68cdd66fb19ccfc1d9f0, UNASSIGN in 1.9180 sec 2023-04-28 08:54:35,423 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672075423"}]},"ts":"1682672075423"} 2023-04-28 08:54:35,424 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 08:54:35,426 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 08:54:35,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9320 sec 2023-04-28 08:54:35,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-28 08:54:35,604 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-28 08:54:35,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete TestAcidGuarantees 2023-04-28 08:54:35,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:35,606 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:35,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-28 08:54:35,607 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:35,612 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:35,614 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/recovered.edits] 2023-04-28 08:54:35,618 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ac54b7c4b8074a11b2ea618596671db0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/ac54b7c4b8074a11b2ea618596671db0 2023-04-28 08:54:35,619 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/d65b977ff6a34cd28e7585a056dc7316 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/d65b977ff6a34cd28e7585a056dc7316 2023-04-28 08:54:35,620 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/db029792a3c34648b1fd13ce5c05048a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/db029792a3c34648b1fd13ce5c05048a 2023-04-28 08:54:35,621 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e1ee331fb7384b1f9230dfc024430d6f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/A/e1ee331fb7384b1f9230dfc024430d6f 2023-04-28 08:54:35,623 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2436b4a7890d45348e939fb8307f8052 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/2436b4a7890d45348e939fb8307f8052 2023-04-28 08:54:35,625 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/5a5cdd0098754619b544c985f2c989c4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/5a5cdd0098754619b544c985f2c989c4 2023-04-28 08:54:35,626 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/82795345252f4d6a833e44223bee8213 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/82795345252f4d6a833e44223bee8213 2023-04-28 08:54:35,627 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d29c6a7cae7143b5a0e9bb6424d27585 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/B/d29c6a7cae7143b5a0e9bb6424d27585 2023-04-28 08:54:35,629 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/0a669f7ea0bf42e792b1acf87d9d7cc5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/0a669f7ea0bf42e792b1acf87d9d7cc5 2023-04-28 08:54:35,630 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2df1b850ee00409b85f072f42cc9c76c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/2df1b850ee00409b85f072f42cc9c76c 2023-04-28 08:54:35,631 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5e9ac3108aef4ad9bbf8afd00c7be372 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/5e9ac3108aef4ad9bbf8afd00c7be372 2023-04-28 08:54:35,632 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f50adfb90e342e2b23195282c06dc78 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/C/8f50adfb90e342e2b23195282c06dc78 2023-04-28 08:54:35,635 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/recovered.edits/363.seqid to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0/recovered.edits/363.seqid 2023-04-28 08:54:35,635 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/e4dc6da0d5be68cdd66fb19ccfc1d9f0 2023-04-28 08:54:35,636 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:54:35,636 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 08:54:35,637 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 08:54:35,640 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280388f7e13a954c3cbb22efc858aa000a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280388f7e13a954c3cbb22efc858aa000a 2023-04-28 08:54:35,641 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280860c3837bb44533b57096d388369a12 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280860c3837bb44533b57096d388369a12 2023-04-28 08:54:35,642 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042828a0a102ce534b009ef29b2b155aadc9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042828a0a102ce534b009ef29b2b155aadc9 2023-04-28 08:54:35,643 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283878f5c9f6924438972c3ed6585d10cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283878f5c9f6924438972c3ed6585d10cb 2023-04-28 08:54:35,644 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428392bc96951084f5d8b0aebdd26597633 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428392bc96951084f5d8b0aebdd26597633 2023-04-28 08:54:35,645 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283a7909343a954437a3e4c2f6e4b687bc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283a7909343a954437a3e4c2f6e4b687bc 2023-04-28 08:54:35,646 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283d9ca24103d24e00a009038e2cc4323e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283d9ca24103d24e00a009038e2cc4323e 2023-04-28 08:54:35,647 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428662a66e35c6f4db8af504074c54d46d3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428662a66e35c6f4db8af504074c54d46d3 2023-04-28 08:54:35,648 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286fe18e29f1b14c13846ebd4b257e2d04 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304286fe18e29f1b14c13846ebd4b257e2d04 2023-04-28 08:54:35,649 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287b1675f8eecc4bcfb206d9501cc50d77 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287b1675f8eecc4bcfb206d9501cc50d77 2023-04-28 08:54:35,650 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042881918221f6c940dcb0761d8a54058f58 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042881918221f6c940dcb0761d8a54058f58 2023-04-28 08:54:35,651 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428844686e79adc44d4a1b01fb7a476b5b9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428844686e79adc44d4a1b01fb7a476b5b9 2023-04-28 08:54:35,653 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042886c5df9b39b941b29afe19059f891fc9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042886c5df9b39b941b29afe19059f891fc9 2023-04-28 08:54:35,654 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289d0a21cdd15442c881dacc4253ad315c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289d0a21cdd15442c881dacc4253ad315c 2023-04-28 08:54:35,655 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a88f839aa840423fb218c23fadded6b6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a88f839aa840423fb218c23fadded6b6 2023-04-28 08:54:35,656 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aa6800cbf1a14bedb67d0f5a042bd58d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aa6800cbf1a14bedb67d0f5a042bd58d 2023-04-28 08:54:35,657 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b9bb6bfa87b7444899d8570e249997c0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b9bb6bfa87b7444899d8570e249997c0 2023-04-28 08:54:35,658 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bda942b8d338422d9d80868618b5cbc9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bda942b8d338422d9d80868618b5cbc9 2023-04-28 08:54:35,659 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dbc0452836b84a788d3d0601267f538d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428dbc0452836b84a788d3d0601267f538d 2023-04-28 08:54:35,660 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e8e08bfb792b4571bf21e909412580af to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e8e08bfb792b4571bf21e909412580af 2023-04-28 08:54:35,661 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ec882abf1fe14e7e8c42b1cb5e1d2785 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ec882abf1fe14e7e8c42b1cb5e1d2785 2023-04-28 08:54:35,662 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ee568a4cba524d6797d14c46367e7f3b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ee568a4cba524d6797d14c46367e7f3b 2023-04-28 08:54:35,663 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 08:54:35,665 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:35,667 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 08:54:35,669 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 08:54:35,670 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:35,670 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 08:54:35,670 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682672075670"}]},"ts":"9223372036854775807"} 2023-04-28 08:54:35,671 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 08:54:35,671 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e4dc6da0d5be68cdd66fb19ccfc1d9f0, NAME => 'TestAcidGuarantees,,1682672048633.e4dc6da0d5be68cdd66fb19ccfc1d9f0.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 08:54:35,671 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 08:54:35,672 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682672075671"}]},"ts":"9223372036854775807"} 2023-04-28 08:54:35,673 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 08:54:35,674 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:35,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 70 msec 2023-04-28 08:54:35,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-28 08:54:35,708 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-28 08:54:35,723 INFO [Listener at localhost/39717] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=236 (was 234) - Thread LEAK? -, OpenFileDescriptor=555 (was 593), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=436 (was 469), ProcessCount=172 (was 172), AvailableMemoryMB=6110 (was 6240) 2023-04-28 08:54:35,738 INFO [Listener at localhost/39717] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=236, OpenFileDescriptor=555, MaxFileDescriptor=60000, SystemLoadAverage=436, ProcessCount=172, AvailableMemoryMB=6109 2023-04-28 08:54:35,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:54:35,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:54:35,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:35,743 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 08:54:35,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-28 08:54:35,744 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 08:54:35,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 08:54:35,745 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:35,745 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b empty. 2023-04-28 08:54:35,746 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:35,746 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:54:35,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 08:54:36,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 08:54:36,153 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:54:36,155 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4def4a766691cce8b483578b280ed61b, NAME => 'TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp 2023-04-28 08:54:36,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 08:54:36,561 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:54:36,561 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4def4a766691cce8b483578b280ed61b, disabling compactions & flushes 2023-04-28 08:54:36,561 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,561 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,561 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. after waiting 0 ms 2023-04-28 08:54:36,561 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,561 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,561 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:36,564 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 08:54:36,565 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672076564"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672076564"}]},"ts":"1682672076564"} 2023-04-28 08:54:36,566 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 08:54:36,566 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 08:54:36,567 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672076566"}]},"ts":"1682672076566"} 2023-04-28 08:54:36,567 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 08:54:36,570 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4def4a766691cce8b483578b280ed61b, ASSIGN}] 2023-04-28 08:54:36,571 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4def4a766691cce8b483578b280ed61b, ASSIGN 2023-04-28 08:54:36,572 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4def4a766691cce8b483578b280ed61b, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=false 2023-04-28 08:54:36,723 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=4def4a766691cce8b483578b280ed61b, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:36,723 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672076723"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672076723"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672076723"}]},"ts":"1682672076723"} 2023-04-28 08:54:36,724 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:54:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 08:54:36,879 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4def4a766691cce8b483578b280ed61b, NAME => 'TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:54:36,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:54:36,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,881 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,882 DEBUG [StoreOpener-4def4a766691cce8b483578b280ed61b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A 2023-04-28 08:54:36,882 DEBUG [StoreOpener-4def4a766691cce8b483578b280ed61b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A 2023-04-28 08:54:36,883 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:36,883 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4def4a766691cce8b483578b280ed61b columnFamilyName A 2023-04-28 08:54:36,883 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.HStore(310): Store=4def4a766691cce8b483578b280ed61b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:36,884 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,885 DEBUG [StoreOpener-4def4a766691cce8b483578b280ed61b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B 2023-04-28 08:54:36,885 DEBUG [StoreOpener-4def4a766691cce8b483578b280ed61b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B 2023-04-28 08:54:36,885 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:36,885 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4def4a766691cce8b483578b280ed61b columnFamilyName B 2023-04-28 08:54:36,886 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.HStore(310): Store=4def4a766691cce8b483578b280ed61b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:36,886 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,887 DEBUG [StoreOpener-4def4a766691cce8b483578b280ed61b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C 2023-04-28 08:54:36,887 DEBUG [StoreOpener-4def4a766691cce8b483578b280ed61b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C 2023-04-28 08:54:36,887 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:54:36,887 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4def4a766691cce8b483578b280ed61b columnFamilyName C 2023-04-28 08:54:36,888 INFO [StoreOpener-4def4a766691cce8b483578b280ed61b-1] regionserver.HStore(310): Store=4def4a766691cce8b483578b280ed61b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:54:36,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:54:36,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:36,893 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:54:36,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 4def4a766691cce8b483578b280ed61b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71623002, jitterRate=0.06726589798927307}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:54:36,894 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:36,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., pid=30, masterSystemTime=1682672076876 2023-04-28 08:54:36,896 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,896 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:36,897 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=4def4a766691cce8b483578b280ed61b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:36,897 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672076897"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672076897"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672076897"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672076897"}]},"ts":"1682672076897"} 2023-04-28 08:54:36,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-28 08:54:36,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 in 174 msec 2023-04-28 08:54:36,901 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-28 08:54:36,901 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4def4a766691cce8b483578b280ed61b, ASSIGN in 329 msec 2023-04-28 08:54:36,902 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 08:54:36,902 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672076902"}]},"ts":"1682672076902"} 2023-04-28 08:54:36,903 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 08:54:36,906 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 08:54:36,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-04-28 08:54:37,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-28 08:54:37,848 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-28 08:54:37,849 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,854 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,855 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,856 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,857 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 08:54:37,859 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:43332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 08:54:37,860 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,864 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,864 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,868 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,869 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,873 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,873 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,876 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,876 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,879 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,879 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,883 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,883 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,888 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,888 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,892 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,893 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,898 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,898 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:54:37,901 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:54:37,905 DEBUG [hconnection-0x72d60ba3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,905 DEBUG [hconnection-0x44867287-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,905 DEBUG [hconnection-0x195479aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,905 DEBUG [hconnection-0x45dc9b18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,906 DEBUG [hconnection-0x3c333ca0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,906 DEBUG [hconnection-0x280bcf4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,907 DEBUG [hconnection-0x1fe93ee5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,907 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,907 DEBUG [hconnection-0x3457eea8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,907 DEBUG [hconnection-0x194fa333-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,907 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,907 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,912 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47108, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,912 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,913 DEBUG [hconnection-0x492f1e6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:54:37,915 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,919 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,919 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,923 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47154, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:54:37,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:37,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:37,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:37,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:37,931 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:37,931 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:37,931 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:37,931 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:37,932 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,932 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:37,932 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:54:37,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:37,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:37,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:37,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:37,934 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:37,934 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:37,934 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,934 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,934 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:37,934 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:37,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:37,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:37,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:37,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:37,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:37,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:37,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:37,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:37,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:37,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:37,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:37,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:37,941 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:37,941 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:37,941 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:37,941 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:37,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:37,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:37,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:37,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:37,943 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,943 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:37,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/bb14182abfb643be9dc4715a5ba82e2f 2023-04-28 08:54:37,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:37,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:37,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725466dd[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:37,946 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:37,946 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,947 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,947 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,947 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:37,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:37,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:37,948 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:37,948 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:37,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:37,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:37,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:37,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:37,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:37,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:37,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:37,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672137947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672137947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:37,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:37,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:37,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672137949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:37,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:37,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672137949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:37,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672137950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:37,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:37,954 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:37,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:37,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:37,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:37,954 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:37,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:37,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:37,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:37,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:37,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:37,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:37,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:37,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:37,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:37,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:37,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:37,964 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:37,964 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:37,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:37,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:37,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:38,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672138051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672138052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672138054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672138056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672138056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,064 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:38,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:38,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:38,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:38,068 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:38,068 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,069 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:38,069 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:38,069 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,069 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:38,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:38,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:38,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:38,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:38,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:38,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:38,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:38,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:38,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:38,072 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:38,072 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:38,072 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:38,073 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:38,073 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:38,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:38,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:38,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:38,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:38,074 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:38,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7aaaad[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:38,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:38,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,078 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:38,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:38,078 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:38,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:38,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:38,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:38,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:38,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:38,082 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:38,082 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:38,082 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:38,082 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:38,082 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:38,084 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,084 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,084 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,084 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:38,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,088 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:38,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:38,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:38,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,088 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:38,088 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:38,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:38,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:38,188 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:38,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:38,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:38,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:38,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:38,192 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:38,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:38,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:38,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,199 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:38,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:38,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:38,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:38,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:38,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:38,201 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:38,201 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:38,201 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:38,201 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:38,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:38,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,203 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,204 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,204 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:38,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@191de472[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:38,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:38,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:38,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:38,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:38,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:38,208 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,208 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:38,208 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:38,208 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:38,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:38,210 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:38,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:38,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:38,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:38,210 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:38,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:38,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,213 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:38,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:38,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:38,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:38,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:38,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:38,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,218 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:38,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,218 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:38,218 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:38,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672138254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672138255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672138257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672138260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672138260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,318 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:38,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:38,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:38,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:38,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:38,322 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:38,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:38,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:38,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,323 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:38,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:38,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:38,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:38,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:38,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:38,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:38,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:38,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:38,325 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:38,325 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:38,325 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:38,325 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:38,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:38,327 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:38,327 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,327 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:38,327 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:38,328 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,329 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,329 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,329 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:38,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de8e15b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:38,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:38,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,333 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,333 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:38,333 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:38,333 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:38,333 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,333 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:38,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,334 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,334 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:38,334 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:38,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:38,336 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:38,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:38,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:38,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:38,337 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:38,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:38,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:38,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,340 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:38,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:38,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:38,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:38,342 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:38,342 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:38,342 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:38,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:38,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:38,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:38,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/9c89dad6eba34af181cc54fc1823213f 2023-04-28 08:54:38,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/666ee73b41ad4748b1590e0b00f3d1fb 2023-04-28 08:54:38,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/bb14182abfb643be9dc4715a5ba82e2f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/bb14182abfb643be9dc4715a5ba82e2f 2023-04-28 08:54:38,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/bb14182abfb643be9dc4715a5ba82e2f, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 08:54:38,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/9c89dad6eba34af181cc54fc1823213f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9c89dad6eba34af181cc54fc1823213f 2023-04-28 08:54:38,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9c89dad6eba34af181cc54fc1823213f, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 08:54:38,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/666ee73b41ad4748b1590e0b00f3d1fb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/666ee73b41ad4748b1590e0b00f3d1fb 2023-04-28 08:54:38,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/666ee73b41ad4748b1590e0b00f3d1fb, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 08:54:38,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4def4a766691cce8b483578b280ed61b in 479ms, sequenceid=15, compaction requested=false 2023-04-28 08:54:38,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:38,442 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:38,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:38,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:38,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:38,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:38,445 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:38,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:38,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:38,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:38,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:38,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:38,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:38,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:38,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:38,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:38,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:38,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:38,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:38,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:38,453 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:38,453 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:38,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:38,560 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:38,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672138589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672138591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672138591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672138592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672138592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672138692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672138696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672138696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672138696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672138696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,865 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/2bbb77e6fab44789ae23c80a49b610e6 2023-04-28 08:54:38,877 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/99bc39d1e94140288c5275856dfeeac3 2023-04-28 08:54:38,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672138896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672138898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672138899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672138899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:38,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:38,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672138899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,200 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672139199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672139200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672139201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672139204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672139204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,287 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e54a83fe66bc4c3db5f9682e68ed2559 2023-04-28 08:54:39,292 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/2bbb77e6fab44789ae23c80a49b610e6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/2bbb77e6fab44789ae23c80a49b610e6 2023-04-28 08:54:39,296 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/2bbb77e6fab44789ae23c80a49b610e6, entries=150, sequenceid=33, filesize=11.6 K 2023-04-28 08:54:39,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/99bc39d1e94140288c5275856dfeeac3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/99bc39d1e94140288c5275856dfeeac3 2023-04-28 08:54:39,303 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/99bc39d1e94140288c5275856dfeeac3, entries=150, sequenceid=33, filesize=11.6 K 2023-04-28 08:54:39,304 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e54a83fe66bc4c3db5f9682e68ed2559 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e54a83fe66bc4c3db5f9682e68ed2559 2023-04-28 08:54:39,308 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e54a83fe66bc4c3db5f9682e68ed2559, entries=150, sequenceid=33, filesize=11.6 K 2023-04-28 08:54:39,308 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for 4def4a766691cce8b483578b280ed61b in 855ms, sequenceid=33, compaction requested=false 2023-04-28 08:54:39,308 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:39,308 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,308 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:39,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:39,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:39,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:39,310 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,310 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,310 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,310 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:39,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cdb1eb5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:39,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:39,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,313 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:39,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:39,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:39,313 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,313 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:39,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,314 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,314 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,315 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:39,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:39,317 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:39,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:39,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:39,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:39,317 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:39,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,321 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:39,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:39,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,324 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:39,325 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:39,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:39,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,425 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:39,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:39,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:39,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:39,428 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:39,428 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,428 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:39,428 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:39,429 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,429 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,430 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,431 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,431 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:39,431 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:39,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:39,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:39,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:39,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:39,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:39,432 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,432 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:39,433 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:54:39,433 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:39,433 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:39,433 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:39,433 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:39,433 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:39,433 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:39,440 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/efc85331608d4e72b736e15d5dcd8302 2023-04-28 08:54:39,451 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/46828a4fb51c4d62b015d7bd876f78bd 2023-04-28 08:54:39,467 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/baac4201786548c09dd77fa2f2e1ec2b 2023-04-28 08:54:39,474 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/efc85331608d4e72b736e15d5dcd8302 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/efc85331608d4e72b736e15d5dcd8302 2023-04-28 08:54:39,483 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/efc85331608d4e72b736e15d5dcd8302, entries=150, sequenceid=49, filesize=11.6 K 2023-04-28 08:54:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/46828a4fb51c4d62b015d7bd876f78bd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/46828a4fb51c4d62b015d7bd876f78bd 2023-04-28 08:54:39,492 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/46828a4fb51c4d62b015d7bd876f78bd, entries=150, sequenceid=49, filesize=11.6 K 2023-04-28 08:54:39,492 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/baac4201786548c09dd77fa2f2e1ec2b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/baac4201786548c09dd77fa2f2e1ec2b 2023-04-28 08:54:39,497 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/baac4201786548c09dd77fa2f2e1ec2b, entries=150, sequenceid=49, filesize=11.6 K 2023-04-28 08:54:39,497 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 4def4a766691cce8b483578b280ed61b in 64ms, sequenceid=49, compaction requested=true 2023-04-28 08:54:39,497 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:39,497 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:39,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:39,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:39,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:39,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,507 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:39,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625aea41[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:39,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:39,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:39,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:39,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:39,512 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,512 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:39,514 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,514 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:39,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:39,517 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:39,517 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:39,517 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:39,517 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:39,517 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:39,519 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,519 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,519 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,520 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:39,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:39,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:39,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,524 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:39,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,524 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:39,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,624 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:39,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:39,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:39,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:39,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:39,628 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:39,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:39,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:39,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:39,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:39,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:39,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:39,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:39,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:39,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:39,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:39,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:39,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:39,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:39,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:39,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:39,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:39,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:39,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1492ac0c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:39,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:39,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:39,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:39,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:39,638 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,638 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:39,639 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,639 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:39,639 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,639 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:39,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:39,642 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:39,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:39,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:39,642 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:39,643 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:39,644 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,644 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,644 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,645 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,649 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:39,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:39,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,650 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:39,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,650 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:39,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:39,711 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:54:39,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:39,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:39,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:39,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:39,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:39,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:39,721 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/79e856712c3541a9b6f99982b85d1d4b 2023-04-28 08:54:39,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672139725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672139726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672139726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672139727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672079721, max timestamp=1682672079727Num uniques 859; 2023-04-28 08:54:39,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672079721, max timestamp=1682672079727Num uniques 859; 2023-04-28 08:54:39,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:54:39,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672079721, max timestamp=1682672079727Num uniques 859; 2023-04-28 08:54:39,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:54:39,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:54:39,750 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:39,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:39,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:39,763 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:39,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:39,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:39,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,766 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:39,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:39,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:39,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:39,768 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,768 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:39,768 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:39,768 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:39,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:39,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:39,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:39,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eaa65fc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:39,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:39,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:39,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:39,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:39,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:39,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,774 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:39,774 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,774 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672139772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:39,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:39,777 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:39,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:39,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:39,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:39,777 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:39,779 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,779 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,780 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,781 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:39,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:39,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,784 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,784 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:39,784 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,784 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:39,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:39,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,784 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672139830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672139830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672139831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672139831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:39,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672139876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,884 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:39,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:39,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:39,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:39,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:39,887 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:39,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:39,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:39,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,889 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,889 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,889 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:39,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:39,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:39,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:39,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:39,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:39,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:39,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:39,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:39,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:39,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:39,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:39,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:39,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:39,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:39,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:39,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,892 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,893 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,893 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:39,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:39,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a90f42a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:39,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,899 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:39,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:39,899 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:39,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:39,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,900 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:39,900 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,900 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:39,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:39,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:39,903 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:39,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:39,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:39,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:39,903 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:39,905 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,905 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,905 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:39,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:39,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:39,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,910 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,910 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,910 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:39,910 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:39,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:39,910 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:39,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:39,910 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:39,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:39,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:39,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:39,911 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:39,911 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:39,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:39,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:40,011 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:40,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:40,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:40,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:40,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:40,015 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:40,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:40,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:40,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:40,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:40,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:40,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:40,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:40,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:40,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:40,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:40,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:40,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:40,018 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,018 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:40,019 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:40,019 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:40,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:40,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:40,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:40,020 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,020 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:40,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2d15db[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:40,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:40,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:40,023 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,023 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:40,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:40,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:40,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:40,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:40,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:40,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:40,028 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:40,028 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:40,028 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:40,028 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:40,028 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:40,030 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,030 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,030 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,030 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,031 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,033 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,033 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,034 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:40,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:40,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:40,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:40,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:40,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:40,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:40,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:40,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,034 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:40,034 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:40,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:40,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:40,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672140033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672140033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672140034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672140034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672140080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,134 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:40,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:40,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:40,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/7c985c9ad1b34e98969852890b41318e 2023-04-28 08:54:40,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:40,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:40,139 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:40,139 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,139 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:40,139 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:40,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,141 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,141 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:40,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:40,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:40,142 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,142 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:40,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:40,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:40,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:40,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:40,143 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:40,143 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,143 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:40,143 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:40,143 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:40,143 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,143 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:40,143 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:40,144 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:40,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:40,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:40,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:40,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:40,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:40,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ede36fe[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:40,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:40,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,150 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:40,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:40,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:40,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:40,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/5a92c1b3e1834c7aac2334a990c120c5 2023-04-28 08:54:40,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:40,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:40,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/79e856712c3541a9b6f99982b85d1d4b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/79e856712c3541a9b6f99982b85d1d4b 2023-04-28 08:54:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:40,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:40,162 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:40,162 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:40,162 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:40,162 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:40,162 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:40,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/79e856712c3541a9b6f99982b85d1d4b, entries=100, sequenceid=61, filesize=9.3 K 2023-04-28 08:54:40,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:40,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,164 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:40,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:40,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:40,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:40,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:40,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:40,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:40,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:40,170 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:40,170 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:40,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/7c985c9ad1b34e98969852890b41318e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7c985c9ad1b34e98969852890b41318e 2023-04-28 08:54:40,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7c985c9ad1b34e98969852890b41318e, entries=100, sequenceid=61, filesize=9.3 K 2023-04-28 08:54:40,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/5a92c1b3e1834c7aac2334a990c120c5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5a92c1b3e1834c7aac2334a990c120c5 2023-04-28 08:54:40,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5a92c1b3e1834c7aac2334a990c120c5, entries=100, sequenceid=61, filesize=9.3 K 2023-04-28 08:54:40,186 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4def4a766691cce8b483578b280ed61b in 475ms, sequenceid=61, compaction requested=true 2023-04-28 08:54:40,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:40,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:40,187 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:40,187 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:40,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:40,187 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:40,188 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:40,188 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:40,188 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,189 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9c89dad6eba34af181cc54fc1823213f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/99bc39d1e94140288c5275856dfeeac3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/46828a4fb51c4d62b015d7bd876f78bd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7c985c9ad1b34e98969852890b41318e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=44.1 K 2023-04-28 08:54:40,189 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:40,189 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:40,189 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,189 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/bb14182abfb643be9dc4715a5ba82e2f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/2bbb77e6fab44789ae23c80a49b610e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/efc85331608d4e72b736e15d5dcd8302, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/79e856712c3541a9b6f99982b85d1d4b] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=44.1 K 2023-04-28 08:54:40,189 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9c89dad6eba34af181cc54fc1823213f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682672077927 2023-04-28 08:54:40,190 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bb14182abfb643be9dc4715a5ba82e2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682672077927 2023-04-28 08:54:40,190 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 99bc39d1e94140288c5275856dfeeac3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682672077946 2023-04-28 08:54:40,190 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 2bbb77e6fab44789ae23c80a49b610e6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682672077946 2023-04-28 08:54:40,191 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 46828a4fb51c4d62b015d7bd876f78bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682672078562 2023-04-28 08:54:40,191 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting efc85331608d4e72b736e15d5dcd8302, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682672078562 2023-04-28 08:54:40,191 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 79e856712c3541a9b6f99982b85d1d4b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682672079710 2023-04-28 08:54:40,192 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7c985c9ad1b34e98969852890b41318e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682672079710 2023-04-28 08:54:40,201 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:40,208 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#210 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:40,270 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:40,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:40,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:40,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:40,277 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:40,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:40,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:40,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,278 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:40,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:40,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:40,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:40,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:40,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:40,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:40,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:40,280 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:40,281 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-28 08:54:40,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:40,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:40,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:40,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:40,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:40,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:40,297 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7fe861414d5e4b039e4cd7c4cec0f967 2023-04-28 08:54:40,314 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/44647528fe4a407a8138b59b4493a826 2023-04-28 08:54:40,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:40,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:40,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672140347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672140347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672140348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672140349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672140384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672140451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672140451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672140452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672140452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,623 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/b9a21dda1c084a03b92fd1a2abff49a4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b9a21dda1c084a03b92fd1a2abff49a4 2023-04-28 08:54:40,632 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into b9a21dda1c084a03b92fd1a2abff49a4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:40,632 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:40,632 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=12, startTime=1682672080187; duration=0sec 2023-04-28 08:54:40,632 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:40,632 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:54:40,633 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:54:40,634 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:40,634 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,634 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/666ee73b41ad4748b1590e0b00f3d1fb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e54a83fe66bc4c3db5f9682e68ed2559, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/baac4201786548c09dd77fa2f2e1ec2b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5a92c1b3e1834c7aac2334a990c120c5] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=44.1 K 2023-04-28 08:54:40,634 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 666ee73b41ad4748b1590e0b00f3d1fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682672077927 2023-04-28 08:54:40,635 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e54a83fe66bc4c3db5f9682e68ed2559, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682672077946 2023-04-28 08:54:40,635 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting baac4201786548c09dd77fa2f2e1ec2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1682672078562 2023-04-28 08:54:40,635 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5a92c1b3e1834c7aac2334a990c120c5, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682672079710 2023-04-28 08:54:40,642 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:40,645 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/3ffdf07b2853480db74837102c52f43e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3ffdf07b2853480db74837102c52f43e 2023-04-28 08:54:40,650 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into 3ffdf07b2853480db74837102c52f43e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:40,650 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:40,650 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=12, startTime=1682672080187; duration=0sec 2023-04-28 08:54:40,650 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:40,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672140655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672140655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672140655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672140655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,732 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/7162f7f4dcde48bfab5d8a0bf8ac8713 2023-04-28 08:54:40,736 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7fe861414d5e4b039e4cd7c4cec0f967 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7fe861414d5e4b039e4cd7c4cec0f967 2023-04-28 08:54:40,741 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7fe861414d5e4b039e4cd7c4cec0f967, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 08:54:40,742 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/44647528fe4a407a8138b59b4493a826 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/44647528fe4a407a8138b59b4493a826 2023-04-28 08:54:40,746 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/44647528fe4a407a8138b59b4493a826, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 08:54:40,747 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/7162f7f4dcde48bfab5d8a0bf8ac8713 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7162f7f4dcde48bfab5d8a0bf8ac8713 2023-04-28 08:54:40,750 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7162f7f4dcde48bfab5d8a0bf8ac8713, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 08:54:40,751 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 4def4a766691cce8b483578b280ed61b in 470ms, sequenceid=95, compaction requested=false 2023-04-28 08:54:40,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:40,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:40,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:40,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:40,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:40,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,753 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:40,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:40,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73462b5c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:40,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:40,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,757 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:40,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:40,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:40,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:40,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:40,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:40,766 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:40,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:40,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:40,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:40,767 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:40,769 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,769 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:40,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,769 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:40,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:40,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:40,771 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:40,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:40,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:40,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:40,774 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:40,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:40,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:40,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,775 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,775 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,775 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:40,775 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:40,775 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:40,775 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:40,775 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:40,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:40,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:40,875 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:40,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:40,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:40,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:40,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:40,878 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:40,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:40,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:40,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,880 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:40,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:40,881 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,881 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:40,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:40,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:40,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:40,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:40,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:40,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:40,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:40,882 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:40,882 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:40,883 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:54:40,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:40,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:40,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:40,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:40,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:40,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:40,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:40,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:40,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672080917, max timestamp=1682672080937Num uniques 859; 2023-04-28 08:54:40,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672080917, max timestamp=1682672080937Num uniques 859; 2023-04-28 08:54:40,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:54:40,939 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:54:40,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672080917, max timestamp=1682672080937Num uniques 859; 2023-04-28 08:54:40,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:54:40,945 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 08:54:40,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672140965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672140968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672140969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672140969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:40,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:40,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672140970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,051 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/6c4ad65df25c40acbdb6faaa71c99469 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6c4ad65df25c40acbdb6faaa71c99469 2023-04-28 08:54:41,058 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into 6c4ad65df25c40acbdb6faaa71c99469(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:41,058 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:41,058 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=12, startTime=1682672080187; duration=0sec 2023-04-28 08:54:41,059 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:41,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672141071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672141074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672141074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672141075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672141075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672141274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672141277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672141277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672141279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672141279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,294 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7877705159904126aa8deb64d38351d9 2023-04-28 08:54:41,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672141577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672141580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672141581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672141581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:41,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672141582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:41,706 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/e92ead9857e046ce96c64242fa788071 2023-04-28 08:54:42,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:42,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672142081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:42,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672142084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:42,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672142085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672142087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672142087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,116 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/561ae051e1e342a39eee552fd0f8a53a 2023-04-28 08:54:42,120 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7877705159904126aa8deb64d38351d9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7877705159904126aa8deb64d38351d9 2023-04-28 08:54:42,124 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7877705159904126aa8deb64d38351d9, entries=150, sequenceid=108, filesize=11.6 K 2023-04-28 08:54:42,125 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/e92ead9857e046ce96c64242fa788071 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e92ead9857e046ce96c64242fa788071 2023-04-28 08:54:42,128 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e92ead9857e046ce96c64242fa788071, entries=150, sequenceid=108, filesize=11.6 K 2023-04-28 08:54:42,129 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/561ae051e1e342a39eee552fd0f8a53a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/561ae051e1e342a39eee552fd0f8a53a 2023-04-28 08:54:42,133 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/561ae051e1e342a39eee552fd0f8a53a, entries=150, sequenceid=108, filesize=11.6 K 2023-04-28 08:54:42,133 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4def4a766691cce8b483578b280ed61b in 1250ms, sequenceid=108, compaction requested=true 2023-04-28 08:54:42,133 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:42,134 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:42,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:42,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:42,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:42,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:42,141 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:42,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eaaa0ba[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:42,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:42,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,145 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,145 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:42,145 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:42,145 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:42,145 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,145 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:42,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:42,146 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:42,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:42,150 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:42,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:42,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:42,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:42,150 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:42,151 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,151 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,151 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:42,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:42,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,157 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:42,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:42,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:42,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:42,158 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:42,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:42,158 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:42,258 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:42,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:42,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:42,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:42,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:42,261 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:42,261 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:42,261 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:42,261 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:42,262 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,262 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,263 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:42,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:42,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:42,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:42,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:42,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:42,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:42,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:42,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:42,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:42,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:42,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:42,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:42,265 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:42,265 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:42,265 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-28 08:54:42,266 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:42,266 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:42,266 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:42,266 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:42,266 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:42,266 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:42,275 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/0fd9294501c445bcabd1a9202c0f2e4b 2023-04-28 08:54:42,305 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/5d4b8ed22d0543209ead952ab7c8c6c0 2023-04-28 08:54:42,723 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/54012042565548bb8cb83bf0cdf53942 2023-04-28 08:54:42,727 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/0fd9294501c445bcabd1a9202c0f2e4b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0fd9294501c445bcabd1a9202c0f2e4b 2023-04-28 08:54:42,730 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0fd9294501c445bcabd1a9202c0f2e4b, entries=150, sequenceid=145, filesize=11.8 K 2023-04-28 08:54:42,731 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/5d4b8ed22d0543209ead952ab7c8c6c0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/5d4b8ed22d0543209ead952ab7c8c6c0 2023-04-28 08:54:42,735 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/5d4b8ed22d0543209ead952ab7c8c6c0, entries=150, sequenceid=145, filesize=11.8 K 2023-04-28 08:54:42,735 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/54012042565548bb8cb83bf0cdf53942 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54012042565548bb8cb83bf0cdf53942 2023-04-28 08:54:42,739 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54012042565548bb8cb83bf0cdf53942, entries=150, sequenceid=145, filesize=11.8 K 2023-04-28 08:54:42,740 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 4def4a766691cce8b483578b280ed61b in 475ms, sequenceid=145, compaction requested=true 2023-04-28 08:54:42,740 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:42,740 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:42,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:42,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:42,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:42,740 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:42,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,741 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,742 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,742 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:42,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29aeeb79[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:42,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:42,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,744 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:42,744 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:42,744 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,744 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:42,744 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:42,747 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:42,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:42,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:42,749 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:42,749 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:42,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:42,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:42,750 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:42,751 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,751 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:42,751 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:42,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:42,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:42,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:42,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:42,756 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,756 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,856 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:42,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:42,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:42,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:42,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:42,860 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:42,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:42,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:42,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:42,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:42,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:42,862 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,862 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:42,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:42,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:42,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:42,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:42,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:42,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:42,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:42,863 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:42,863 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:42,863 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:42,863 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:42,864 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:42,864 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:42,864 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,864 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:42,864 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:42,867 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,867 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,868 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,868 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:42,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d85174[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:42,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:42,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,870 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,870 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:42,870 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:42,870 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:42,870 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,870 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:42,872 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,872 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,872 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:42,872 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:42,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:42,874 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:42,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:42,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:42,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:42,874 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:42,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:42,876 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,884 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:42,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:42,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:42,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:42,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,885 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,885 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:42,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:42,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:42,985 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:42,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:42,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:42,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:42,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:42,989 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:42,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:42,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:42,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:42,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:42,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:42,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:42,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:42,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:42,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:42,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:42,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:42,992 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:42,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:42,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:42,993 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:42,993 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:42,993 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:42,993 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:42,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:42,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:42,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:42,993 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:42,994 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:42,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:42,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:42,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74069038[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:42,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:42,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,998 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:42,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:42,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:42,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:42,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:42,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,000 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,002 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,002 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,002 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,002 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,002 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,004 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,004 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,004 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:43,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,008 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,008 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:43,098 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:54:43,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:43,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:43,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:43,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:43,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:43,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:43,108 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:43,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:43,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:43,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:43,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:43,112 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:43,112 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,112 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:43,112 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:43,112 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,112 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,115 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,115 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,115 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,115 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:43,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:43,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:43,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:43,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:43,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:43,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:43,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:43,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:43,116 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:43,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:43,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:43,117 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:43,118 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,118 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,119 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,119 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:43,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672143115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672083109, max timestamp=1682672083117Num uniques 859; 2023-04-28 08:54:43,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:54:43,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672083109, max timestamp=1682672083117Num uniques 859; 2023-04-28 08:54:43,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:54:43,120 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682672083109, max timestamp=1682672083117Num uniques 859; 2023-04-28 08:54:43,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 08:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd24172[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:43,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:43,123 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,124 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,124 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:43,124 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:43,124 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:43,124 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,124 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,125 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,128 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,128 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,135 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,135 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,135 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,136 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,139 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,139 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,139 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,140 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672143141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672143141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672143143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672143143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672143220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,240 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:43,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:43,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:43,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:43,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:43,243 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:43,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:43,243 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:43,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,244 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,245 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:43,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:43,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:43,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:43,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:43,247 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:43,247 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672143246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672143247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672143247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672143247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,249 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,249 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,249 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:43,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f996d2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:43,251 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:43,251 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,254 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:43,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:43,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:43,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,256 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,258 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,258 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,258 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,259 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,260 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,262 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,268 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:43,269 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,269 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,369 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:43,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:43,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:43,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:43,373 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:43,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:43,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:43,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,378 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:43,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:43,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:43,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:43,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:43,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:43,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:43,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:43,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:43,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:43,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:43,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:43,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,381 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25de2e68[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:43,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:43,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,385 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:43,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:43,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:43,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,386 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,390 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,390 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,393 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,397 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:43,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,397 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,397 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672143425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672143450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672143450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672143450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672143450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,497 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:43,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:43,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:43,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:43,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:43,501 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:43,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:43,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:43,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,503 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,504 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:43,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:43,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:43,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:43,505 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,505 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:43,505 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:43,505 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:43,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:43,507 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,507 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/9aedf56880074aee8a54d76b149869b8 2023-04-28 08:54:43,507 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,507 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:43,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6519edbb[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:43,509 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:43,509 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,510 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:43,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:43,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:43,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,512 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,515 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,515 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,520 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,523 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,523 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,623 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:43,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:43,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:43,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:43,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:43,627 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:43,627 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,627 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:43,627 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:43,627 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,627 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,628 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:43,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:43,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:43,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:43,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:43,631 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:43,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:43,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:43,633 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:43,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785df9bb[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:43,635 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:43,635 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,637 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,637 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:43,637 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:43,637 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:43,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,638 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,641 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,641 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,643 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:43,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,647 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,647 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672143729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,747 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:43,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:43,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:43,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:43,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:43,750 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:43,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:43,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:43,751 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,751 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672143753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:43,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672143754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672143755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:43,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:43,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:43,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:43,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:43,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:43,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:43,757 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,757 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:43,758 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:43,758 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:43,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:43,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:43,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:43,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672143755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,761 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:43,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650fbf64[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:43,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:43,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:43,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:43,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:43,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,765 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,767 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,767 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,771 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:43,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,774 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,774 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,874 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:43,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:43,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:43,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:43,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:43,877 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:43,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:43,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:43,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,879 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:43,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:43,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:43,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:43,881 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,881 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:43,881 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:43,881 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:43,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:43,883 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:43,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f670c3[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:43,884 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:43,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,886 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:43,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:43,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:43,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:43,888 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,888 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:43,888 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,888 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:43,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:43,891 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:43,891 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:43,891 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:43,891 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:43,891 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:43,893 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:43,893 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,893 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:43,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:43,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:43,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:43,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:43,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:43,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:43,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:43,901 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:43,901 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:43,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:43,919 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/6816d294edb3463f987980f9d074757e 2023-04-28 08:54:44,001 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:44,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:44,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:44,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:44,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:44,005 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:44,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:44,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:44,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:44,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:44,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:44,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:44,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:44,009 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:44,009 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,009 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:44,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:44,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:44,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:44,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:44,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:44,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:44,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:44,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:44,011 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,012 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,012 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:44,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ba83773[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:44,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:44,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,015 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:44,015 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:44,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:44,015 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:44,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,016 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,016 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:44,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:44,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:44,019 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:44,019 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:44,019 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:44,019 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:44,019 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:44,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:44,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,023 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:44,025 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:44,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:44,025 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:44,025 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,026 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:44,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:44,126 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:44,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:44,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:44,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:44,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:44,129 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:44,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:44,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:44,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,131 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:44,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:44,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:44,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:44,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:44,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:44,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:44,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:44,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:44,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:44,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:44,133 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,133 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:44,133 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:44,133 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:44,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:44,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:44,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:44,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,135 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,135 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:44,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@caf859c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:44,136 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:44,136 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,140 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,140 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:44,140 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:44,140 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:44,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:44,141 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:44,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:44,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:44,144 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:44,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:44,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:44,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:44,144 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:44,145 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,145 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,145 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:44,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:44,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:44,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:44,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:44,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:44,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:44,150 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:44,150 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,150 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:44,150 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,150 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,150 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:44,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:44,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:44,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672144235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,250 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:44,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:44,253 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:44,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:44,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:44,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,256 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:44,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:44,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:44,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:44,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:44,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:44,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:44,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:44,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:44,258 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,258 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:44,258 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:44,258 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:44,258 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:44,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672144258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:44,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672144258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,259 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1443fff2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:44,261 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:44,262 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:44,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672144259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,263 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:44,263 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:44,263 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:44,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:44,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:44,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672144262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,264 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,264 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:44,264 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,264 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:44,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:44,266 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:44,266 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:44,266 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:44,266 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:44,266 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:44,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:44,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:44,269 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:44,273 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:44,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:44,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:44,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:44,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:44,273 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:44,273 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:44,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/3cd95ebe1adb4b14b38b78e1721380cb 2023-04-28 08:54:44,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/9aedf56880074aee8a54d76b149869b8 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9aedf56880074aee8a54d76b149869b8 2023-04-28 08:54:44,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9aedf56880074aee8a54d76b149869b8, entries=200, sequenceid=158, filesize=14.1 K 2023-04-28 08:54:44,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/6816d294edb3463f987980f9d074757e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6816d294edb3463f987980f9d074757e 2023-04-28 08:54:44,344 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6816d294edb3463f987980f9d074757e, entries=150, sequenceid=158, filesize=11.8 K 2023-04-28 08:54:44,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/3cd95ebe1adb4b14b38b78e1721380cb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/3cd95ebe1adb4b14b38b78e1721380cb 2023-04-28 08:54:44,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/3cd95ebe1adb4b14b38b78e1721380cb, entries=150, sequenceid=158, filesize=11.8 K 2023-04-28 08:54:44,349 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.31 KB/132415 for 4def4a766691cce8b483578b280ed61b in 1251ms, sequenceid=158, compaction requested=true 2023-04-28 08:54:44,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:44,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:44,350 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:44,350 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:44,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:44,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:44,351 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62258 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:44,351 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:44,351 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:44,351 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,351 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:44,351 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3ffdf07b2853480db74837102c52f43e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7fe861414d5e4b039e4cd7c4cec0f967, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7877705159904126aa8deb64d38351d9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0fd9294501c445bcabd1a9202c0f2e4b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9aedf56880074aee8a54d76b149869b8] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=60.8 K 2023-04-28 08:54:44,352 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,352 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b9a21dda1c084a03b92fd1a2abff49a4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/44647528fe4a407a8138b59b4493a826, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e92ead9857e046ce96c64242fa788071, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/5d4b8ed22d0543209ead952ab7c8c6c0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6816d294edb3463f987980f9d074757e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=58.5 K 2023-04-28 08:54:44,352 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 3ffdf07b2853480db74837102c52f43e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682672078591 2023-04-28 08:54:44,352 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b9a21dda1c084a03b92fd1a2abff49a4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682672078591 2023-04-28 08:54:44,352 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7fe861414d5e4b039e4cd7c4cec0f967, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672079751 2023-04-28 08:54:44,353 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 44647528fe4a407a8138b59b4493a826, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672079751 2023-04-28 08:54:44,353 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7877705159904126aa8deb64d38351d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682672080343 2023-04-28 08:54:44,353 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e92ead9857e046ce96c64242fa788071, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682672080343 2023-04-28 08:54:44,353 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 0fd9294501c445bcabd1a9202c0f2e4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682672080967 2023-04-28 08:54:44,354 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5d4b8ed22d0543209ead952ab7c8c6c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682672080967 2023-04-28 08:54:44,354 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9aedf56880074aee8a54d76b149869b8, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672083095 2023-04-28 08:54:44,354 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6816d294edb3463f987980f9d074757e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672083095 2023-04-28 08:54:44,364 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#224 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:44,365 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:44,374 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:44,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:44,377 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/8a03e98fbb684f30955d5ff830e9f416 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8a03e98fbb684f30955d5ff830e9f416 2023-04-28 08:54:44,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:44,381 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into 8a03e98fbb684f30955d5ff830e9f416(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:44,381 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:44,381 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=11, startTime=1682672084350; duration=0sec 2023-04-28 08:54:44,382 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:44,382 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:44,383 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59868 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:44,383 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:44,387 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:44,388 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6c4ad65df25c40acbdb6faaa71c99469, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7162f7f4dcde48bfab5d8a0bf8ac8713, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/561ae051e1e342a39eee552fd0f8a53a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54012042565548bb8cb83bf0cdf53942, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/3cd95ebe1adb4b14b38b78e1721380cb] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=58.5 K 2023-04-28 08:54:44,388 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:44,388 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,388 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:44,388 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:44,388 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6c4ad65df25c40acbdb6faaa71c99469, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1682672078591 2023-04-28 08:54:44,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,389 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7162f7f4dcde48bfab5d8a0bf8ac8713, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672079751 2023-04-28 08:54:44,389 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 561ae051e1e342a39eee552fd0f8a53a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1682672080343 2023-04-28 08:54:44,390 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 54012042565548bb8cb83bf0cdf53942, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1682672080967 2023-04-28 08:54:44,390 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3cd95ebe1adb4b14b38b78e1721380cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672083095 2023-04-28 08:54:44,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:44,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:44,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:44,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:44,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:44,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:44,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:44,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:44,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:44,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:44,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:44,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:44,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:44,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:44,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:44,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:44,394 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=129.31 KB heapSize=369.23 KB 2023-04-28 08:54:44,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:44,395 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:44,395 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:44,395 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:44,395 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:44,395 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:54:44,395 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/a9e91bb864a94d0596fa6a34752bc41e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/a9e91bb864a94d0596fa6a34752bc41e 2023-04-28 08:54:44,405 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#227 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:44,413 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into a9e91bb864a94d0596fa6a34752bc41e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:44,413 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:44,413 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=11, startTime=1682672084350; duration=0sec 2023-04-28 08:54:44,413 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:44,413 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/4baa38d611844bde8edadfcf18a7e523 2023-04-28 08:54:44,431 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/6343bd49cb8b4487bc126b047f5bb60c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6343bd49cb8b4487bc126b047f5bb60c 2023-04-28 08:54:44,441 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into 6343bd49cb8b4487bc126b047f5bb60c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:44,441 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:44,441 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=11, startTime=1682672084350; duration=0sec 2023-04-28 08:54:44,441 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:44,844 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/14a9e36fc75442028f87307f61ca0ede 2023-04-28 08:54:45,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:45,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:45,255 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/54dec07001684e2ca63523a44dff36c2 2023-04-28 08:54:45,260 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/4baa38d611844bde8edadfcf18a7e523 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4baa38d611844bde8edadfcf18a7e523 2023-04-28 08:54:45,264 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4baa38d611844bde8edadfcf18a7e523, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 08:54:45,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672145262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,265 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/14a9e36fc75442028f87307f61ca0ede as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/14a9e36fc75442028f87307f61ca0ede 2023-04-28 08:54:45,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672145263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672145267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672145268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672145271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,275 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/14a9e36fc75442028f87307f61ca0ede, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 08:54:45,276 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/54dec07001684e2ca63523a44dff36c2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54dec07001684e2ca63523a44dff36c2 2023-04-28 08:54:45,280 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54dec07001684e2ca63523a44dff36c2, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 08:54:45,281 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.31 KB/132415, heapSize ~369.18 KB/378040, currentSize=53.67 KB/54960 for 4def4a766691cce8b483578b280ed61b in 887ms, sequenceid=195, compaction requested=false 2023-04-28 08:54:45,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:45,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:45,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:45,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:45,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:45,282 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:45,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16b1f5d1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:45,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:45,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,286 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:45,286 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,287 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:45,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:45,287 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:45,288 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,288 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:45,288 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,288 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:45,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:45,291 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:45,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:45,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:45,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:45,291 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:45,293 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,293 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,293 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:45,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:45,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:45,298 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:45,298 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:45,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:45,368 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:45,369 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:45,369 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:45,369 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:45,369 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:45,369 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:45,369 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:45,398 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:45,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:45,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:45,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:45,401 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:45,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:45,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:45,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,404 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:45,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:45,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:45,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:45,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:45,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:45,408 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,408 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:45,408 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:45,408 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:45,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:45,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:45,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:45,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,410 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,410 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:45,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cf84bdb[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:45,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:45,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,414 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:45,414 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:45,414 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:45,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:45,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:45,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:45,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:45,420 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:45,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:45,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:45,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:45,420 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:45,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,422 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672145420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,422 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,422 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672145420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:45,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:45,427 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:45,427 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:45,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,526 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672145523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672145523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,527 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:45,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:45,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:45,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:45,530 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:45,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:45,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:45,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,532 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,532 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,532 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,532 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:45,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:45,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:45,534 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,534 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:45,534 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:45,534 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:45,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:45,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,538 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:45,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c8ba4bd[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:45,539 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:45,539 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,540 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:45,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:45,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:45,541 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,541 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:45,542 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:45,542 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,542 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:45,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:45,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:45,544 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:45,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:45,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:45,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:45,544 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:45,546 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,546 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,546 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,547 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,554 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,554 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,554 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:45,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:45,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,554 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,555 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:45,555 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:45,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,655 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:45,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:45,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:45,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:45,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:45,658 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:45,658 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,658 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:45,659 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:45,659 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,659 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,660 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:45,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:45,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:45,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:45,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:45,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:45,664 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,664 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,664 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,664 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:45,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37bc60b1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:45,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:45,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,667 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,667 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,667 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:45,669 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,670 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,670 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:45,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:45,672 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:45,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:45,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:45,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:45,672 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:45,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,674 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,675 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,677 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,677 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,677 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:45,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:45,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,678 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:45,678 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:45,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47108 deadline: 1682672145727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:45,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672145728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,777 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/0c36c617fef54732a35189764fb51e62 2023-04-28 08:54:45,779 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:45,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:45,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:45,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:45,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:45,783 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:45,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:45,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:45,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:45,788 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:45,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:45,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:45,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:45,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:45,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:45,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:45,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:45,814 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,814 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:45,814 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:45,814 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:45,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:45,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:45,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:45,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,817 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,817 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,817 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e3a04ea[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:45,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:45,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,824 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,824 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:45,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:45,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:45,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:45,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/cf77d4c083cb4cfa94d83529c990895f 2023-04-28 08:54:45,826 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,826 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,826 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,826 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:45,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:45,830 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:45,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:45,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:45,830 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:45,830 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:45,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:45,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,838 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:45,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:45,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:45,843 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:45,844 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,844 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:45,844 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,844 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:45,844 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:45,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:45,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/a21c350faa9645f586b51cbf2d46fd1f 2023-04-28 08:54:45,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/0c36c617fef54732a35189764fb51e62 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0c36c617fef54732a35189764fb51e62 2023-04-28 08:54:45,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0c36c617fef54732a35189764fb51e62, entries=200, sequenceid=210, filesize=14.1 K 2023-04-28 08:54:45,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/cf77d4c083cb4cfa94d83529c990895f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/cf77d4c083cb4cfa94d83529c990895f 2023-04-28 08:54:45,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/cf77d4c083cb4cfa94d83529c990895f, entries=150, sequenceid=210, filesize=11.8 K 2023-04-28 08:54:45,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/a21c350faa9645f586b51cbf2d46fd1f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a21c350faa9645f586b51cbf2d46fd1f 2023-04-28 08:54:45,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a21c350faa9645f586b51cbf2d46fd1f, entries=150, sequenceid=210, filesize=11.8 K 2023-04-28 08:54:45,876 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 4def4a766691cce8b483578b280ed61b in 508ms, sequenceid=210, compaction requested=true 2023-04-28 08:54:45,876 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:45,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:45,876 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:45,876 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:45,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:45,876 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:45,877 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38806 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:45,877 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:45,877 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:45,878 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:45,878 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,878 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,878 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/a9e91bb864a94d0596fa6a34752bc41e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4baa38d611844bde8edadfcf18a7e523, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0c36c617fef54732a35189764fb51e62] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=37.9 K 2023-04-28 08:54:45,878 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8a03e98fbb684f30955d5ff830e9f416, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/14a9e36fc75442028f87307f61ca0ede, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/cf77d4c083cb4cfa94d83529c990895f] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=35.6 K 2023-04-28 08:54:45,878 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a9e91bb864a94d0596fa6a34752bc41e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672083095 2023-04-28 08:54:45,878 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8a03e98fbb684f30955d5ff830e9f416, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672083095 2023-04-28 08:54:45,879 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4baa38d611844bde8edadfcf18a7e523, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672083139 2023-04-28 08:54:45,879 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 14a9e36fc75442028f87307f61ca0ede, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672083139 2023-04-28 08:54:45,879 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 0c36c617fef54732a35189764fb51e62, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682672085255 2023-04-28 08:54:45,879 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting cf77d4c083cb4cfa94d83529c990895f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682672085255 2023-04-28 08:54:45,892 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:45,898 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#234 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:45,944 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:45,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:45,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:45,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:45,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:45,948 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:45,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:45,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:45,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:45,949 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:45,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:45,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:45,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:45,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:45,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:45,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:45,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:45,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:45,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:45,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:45,951 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:45,951 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:45,951 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:45,952 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:45,952 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:45,952 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:45,952 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:45,952 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:45,952 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:45,962 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/e51667bd8d044a989651115ea93c3201 2023-04-28 08:54:45,974 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/9d24ced66ff945ec8abec05d9c97ebaa 2023-04-28 08:54:46,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:46,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:46,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672146046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672146047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672146150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672146150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,321 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/2fb3e402f6fe47fbb2d98ba608b5562b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/2fb3e402f6fe47fbb2d98ba608b5562b 2023-04-28 08:54:46,321 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/da59727f4cae4e49844b0da1845674cb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/da59727f4cae4e49844b0da1845674cb 2023-04-28 08:54:46,326 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into 2fb3e402f6fe47fbb2d98ba608b5562b(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:46,326 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:46,326 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=13, startTime=1682672085876; duration=0sec 2023-04-28 08:54:46,326 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:46,326 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:46,326 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into da59727f4cae4e49844b0da1845674cb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:46,327 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:46,327 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=13, startTime=1682672085876; duration=0sec 2023-04-28 08:54:46,327 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:46,327 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:46,328 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:46,328 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:46,328 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6343bd49cb8b4487bc126b047f5bb60c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54dec07001684e2ca63523a44dff36c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a21c350faa9645f586b51cbf2d46fd1f] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=35.6 K 2023-04-28 08:54:46,328 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6343bd49cb8b4487bc126b047f5bb60c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672083095 2023-04-28 08:54:46,328 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 54dec07001684e2ca63523a44dff36c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672083139 2023-04-28 08:54:46,329 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a21c350faa9645f586b51cbf2d46fd1f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682672085255 2023-04-28 08:54:46,334 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:46,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672146353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672146353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,393 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/a44794e5caa1441f873a6f32434a4d42 2023-04-28 08:54:46,398 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/e51667bd8d044a989651115ea93c3201 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e51667bd8d044a989651115ea93c3201 2023-04-28 08:54:46,403 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e51667bd8d044a989651115ea93c3201, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:54:46,404 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/9d24ced66ff945ec8abec05d9c97ebaa as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9d24ced66ff945ec8abec05d9c97ebaa 2023-04-28 08:54:46,408 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9d24ced66ff945ec8abec05d9c97ebaa, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:54:46,408 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/a44794e5caa1441f873a6f32434a4d42 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a44794e5caa1441f873a6f32434a4d42 2023-04-28 08:54:46,412 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a44794e5caa1441f873a6f32434a4d42, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:54:46,413 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4def4a766691cce8b483578b280ed61b in 462ms, sequenceid=230, compaction requested=false 2023-04-28 08:54:46,413 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:46,413 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:46,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:46,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:46,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:46,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:46,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,415 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:46,415 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,415 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20122465[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:46,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:46,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,418 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:46,418 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:46,418 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:46,418 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,418 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:46,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:46,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:46,420 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:46,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:46,423 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:46,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:46,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:46,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:46,424 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:46,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:46,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:46,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:46,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,429 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:46,429 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:46,429 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:46,429 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:46,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:46,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:46,429 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:46,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:46,430 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:46,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:46,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,430 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:46,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:46,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:46,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,530 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:46,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:46,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:46,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:46,533 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:46,533 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:46,533 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:46,533 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:46,534 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,534 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:46,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:46,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:46,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:46,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:46,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:46,536 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:46,536 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:46,536 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:46,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:46,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:46,537 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:46,537 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:46,545 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/ded396b167ca4df0a317f7244da6f0ff 2023-04-28 08:54:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:46,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:46,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672146681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672146681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,743 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e50e3494117b4334b13bc6d16adb6e71 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e50e3494117b4334b13bc6d16adb6e71 2023-04-28 08:54:46,748 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into e50e3494117b4334b13bc6d16adb6e71(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:46,748 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:46,748 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=13, startTime=1682672085876; duration=0sec 2023-04-28 08:54:46,748 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:46,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672146784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672146784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,977 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/8559608b5a384445bebb56c016b92ad5 2023-04-28 08:54:46,986 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/25eca1c028694effa820a2d7c831eab5 2023-04-28 08:54:46,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672146987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:46,990 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/ded396b167ca4df0a317f7244da6f0ff as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ded396b167ca4df0a317f7244da6f0ff 2023-04-28 08:54:46,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672146988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:46,994 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ded396b167ca4df0a317f7244da6f0ff, entries=150, sequenceid=244, filesize=11.8 K 2023-04-28 08:54:46,995 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/8559608b5a384445bebb56c016b92ad5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8559608b5a384445bebb56c016b92ad5 2023-04-28 08:54:46,998 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8559608b5a384445bebb56c016b92ad5, entries=150, sequenceid=244, filesize=11.8 K 2023-04-28 08:54:46,999 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/25eca1c028694effa820a2d7c831eab5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/25eca1c028694effa820a2d7c831eab5 2023-04-28 08:54:47,003 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/25eca1c028694effa820a2d7c831eab5, entries=150, sequenceid=244, filesize=11.8 K 2023-04-28 08:54:47,004 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4def4a766691cce8b483578b280ed61b in 467ms, sequenceid=244, compaction requested=true 2023-04-28 08:54:47,004 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:47,004 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,004 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:47,004 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:47,004 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,004 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:47,004 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:47,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,007 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,007 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,007 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:47,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@415dc296[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:47,009 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:47,009 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:47,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:47,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:47,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:47,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:47,011 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:47,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:47,014 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:47,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:47,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:47,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:47,014 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:47,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,019 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,023 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:47,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:47,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,023 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:47,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,024 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,024 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,024 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,124 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:47,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:47,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:47,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:47,127 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:47,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:47,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:47,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,129 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:47,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:47,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:47,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:47,131 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:47,131 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:47,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:47,131 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:54:47,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:47,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:47,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:47,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:47,131 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:47,132 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:47,140 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/ab695027fb1b44908f6d4eea1953494a 2023-04-28 08:54:47,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:47,281 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:47,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672147290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672147291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672147293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672147293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672147294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672147395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672147397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672147397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,556 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/f05c4964d9514b6a9fcfc6eddd4509d7 2023-04-28 08:54:47,567 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/7d387fa114b84ca3bcbbddb8542ef6b0 2023-04-28 08:54:47,571 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/ab695027fb1b44908f6d4eea1953494a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ab695027fb1b44908f6d4eea1953494a 2023-04-28 08:54:47,575 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ab695027fb1b44908f6d4eea1953494a, entries=150, sequenceid=265, filesize=11.9 K 2023-04-28 08:54:47,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/f05c4964d9514b6a9fcfc6eddd4509d7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/f05c4964d9514b6a9fcfc6eddd4509d7 2023-04-28 08:54:47,580 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/f05c4964d9514b6a9fcfc6eddd4509d7, entries=150, sequenceid=265, filesize=11.9 K 2023-04-28 08:54:47,580 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/7d387fa114b84ca3bcbbddb8542ef6b0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7d387fa114b84ca3bcbbddb8542ef6b0 2023-04-28 08:54:47,584 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7d387fa114b84ca3bcbbddb8542ef6b0, entries=150, sequenceid=265, filesize=11.9 K 2023-04-28 08:54:47,585 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4def4a766691cce8b483578b280ed61b in 454ms, sequenceid=265, compaction requested=true 2023-04-28 08:54:47,585 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:47,585 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:47,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:47,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:47,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:47,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:47,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15308be4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:47,589 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:47,589 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,590 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:47,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:47,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:47,591 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,591 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:47,593 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,593 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:47,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:47,597 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:47,597 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:47,597 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:47,597 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:47,597 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:47,598 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,598 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,599 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:47,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,600 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,600 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:54:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:47,601 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:47,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:47,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,607 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:47,607 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:47,607 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,607 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/cfbc1a1f1da14b328613ce3997f6b1af 2023-04-28 08:54:47,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672147625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672147625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672147626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,707 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:47,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:47,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:47,710 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:47,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:47,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:47,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,713 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:47,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:47,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:47,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:47,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:47,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:47,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:47,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:47,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:47,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:47,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:47,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:47,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:47,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:47,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,716 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,717 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,717 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:47,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b53c0fc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:47,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:47,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,719 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:47,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:47,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:47,719 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,719 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:47,721 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:47,721 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,721 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,721 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:47,723 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:47,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:47,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:47,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:47,723 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:47,726 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,726 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,726 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,727 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,728 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,730 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:47,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,730 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:47,731 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:47,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:47,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672147730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672147730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672147730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672147795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672147798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,831 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:47,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:47,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:47,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:47,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:47,834 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:47,834 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,834 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:47,834 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:47,834 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,834 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,836 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:47,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:47,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:47,837 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:47,837 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,837 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:47,837 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:47,837 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:47,838 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,838 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:47,838 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:47,838 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,838 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:47,838 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:47,838 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,838 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:47,838 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:47,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,839 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,839 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:47,841 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:47,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad3f0ff[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:47,841 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,843 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:47,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:47,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:47,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,844 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:47,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:47,845 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,845 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:47,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:47,848 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:47,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:47,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:47,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:47,848 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:47,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,852 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:47,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:47,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:47,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,854 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,855 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:47,855 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:47,854 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672147933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672147934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:47,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672147934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,955 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:47,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:47,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:47,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:47,958 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:47,958 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,958 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:47,958 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:47,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,961 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,961 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,961 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,961 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:47,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:47,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:47,963 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,963 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:47,963 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:47,963 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:47,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:47,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:47,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:47,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b5d041[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:47,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,967 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:47,967 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:47,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:47,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:47,969 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:47,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:47,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:47,971 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:47,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:47,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:47,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:47,971 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:47,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:47,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:47,974 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:47,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:47,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:47,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:47,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:47,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:47,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:47,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:47,977 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:47,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:47,978 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:47,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:47,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:47,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:48,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/e7768cc6d43148a4804a50d709ec5f11 2023-04-28 08:54:48,078 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:48,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:48,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:48,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:48,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:48,081 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:48,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:48,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:48,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,082 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:48,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:48,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:48,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:48,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:48,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:48,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:48,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:48,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:48,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@644b0ab8[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:48,096 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:48,096 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,097 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,097 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:48,097 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:48,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:48,097 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:48,099 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:48,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:48,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:48,101 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:48,101 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:48,101 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:48,101 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:48,101 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:48,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:48,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,103 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:48,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:48,107 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:48,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:48,107 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:48,107 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,107 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:48,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:48,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:48,208 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:48,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:48,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:48,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:48,211 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:48,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:48,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:48,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,213 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:48,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:48,213 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,214 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:48,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:48,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:48,215 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,215 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:48,215 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:48,215 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:48,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:48,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11ea72ed[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:48,219 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:48,219 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,220 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,220 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:48,220 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:48,220 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:48,220 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,220 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:48,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:48,225 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:48,228 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:48,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:48,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:48,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:48,228 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:48,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:48,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:48,231 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,232 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,234 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:48,234 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:48,234 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:48,234 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:48,234 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,234 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,234 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,234 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:48,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:48,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:48,235 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:48,235 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:48,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:48,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672148235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672148237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672148238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,335 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:48,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:48,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:48,338 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:48,338 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:48,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:48,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,340 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:48,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:48,340 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:48,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:48,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:48,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:48,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:48,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:48,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:48,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:48,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:48,343 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:48,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:48,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:48,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:48,345 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,345 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,346 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,346 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:48,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:48,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54fb3c27[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:48,349 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:48,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:48,349 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:48,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:48,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:48,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:48,353 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:48,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:48,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:48,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:48,353 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:48,355 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,355 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:48,355 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:48,355 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,356 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:48,360 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:48,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:48,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:48,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,360 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:48,360 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:48,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:48,361 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:48,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:48,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/1f5ee6fe11ea4ea6945674d0b42d2980 2023-04-28 08:54:48,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/cfbc1a1f1da14b328613ce3997f6b1af as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/cfbc1a1f1da14b328613ce3997f6b1af 2023-04-28 08:54:48,442 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/cfbc1a1f1da14b328613ce3997f6b1af, entries=150, sequenceid=278, filesize=11.9 K 2023-04-28 08:54:48,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/e7768cc6d43148a4804a50d709ec5f11 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e7768cc6d43148a4804a50d709ec5f11 2023-04-28 08:54:48,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e7768cc6d43148a4804a50d709ec5f11, entries=150, sequenceid=278, filesize=11.9 K 2023-04-28 08:54:48,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/1f5ee6fe11ea4ea6945674d0b42d2980 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/1f5ee6fe11ea4ea6945674d0b42d2980 2023-04-28 08:54:48,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/1f5ee6fe11ea4ea6945674d0b42d2980, entries=150, sequenceid=278, filesize=11.9 K 2023-04-28 08:54:48,454 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4def4a766691cce8b483578b280ed61b in 854ms, sequenceid=278, compaction requested=true 2023-04-28 08:54:48,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:48,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:48,454 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:48,454 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:48,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:48,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:48,456 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60840 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:48,456 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60840 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:48,456 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:48,456 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:48,457 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,457 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,457 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/da59727f4cae4e49844b0da1845674cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e51667bd8d044a989651115ea93c3201, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ded396b167ca4df0a317f7244da6f0ff, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ab695027fb1b44908f6d4eea1953494a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/cfbc1a1f1da14b328613ce3997f6b1af] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=59.4 K 2023-04-28 08:54:48,457 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/2fb3e402f6fe47fbb2d98ba608b5562b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9d24ced66ff945ec8abec05d9c97ebaa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8559608b5a384445bebb56c016b92ad5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/f05c4964d9514b6a9fcfc6eddd4509d7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e7768cc6d43148a4804a50d709ec5f11] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=59.4 K 2023-04-28 08:54:48,457 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting da59727f4cae4e49844b0da1845674cb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682672085255 2023-04-28 08:54:48,457 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2fb3e402f6fe47fbb2d98ba608b5562b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682672085255 2023-04-28 08:54:48,457 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e51667bd8d044a989651115ea93c3201, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672085411 2023-04-28 08:54:48,457 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9d24ced66ff945ec8abec05d9c97ebaa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672085411 2023-04-28 08:54:48,458 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ded396b167ca4df0a317f7244da6f0ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682672086040 2023-04-28 08:54:48,458 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8559608b5a384445bebb56c016b92ad5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682672086040 2023-04-28 08:54:48,458 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ab695027fb1b44908f6d4eea1953494a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672086662 2023-04-28 08:54:48,458 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting f05c4964d9514b6a9fcfc6eddd4509d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672086662 2023-04-28 08:54:48,459 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting cfbc1a1f1da14b328613ce3997f6b1af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682672087289 2023-04-28 08:54:48,459 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e7768cc6d43148a4804a50d709ec5f11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682672087289 2023-04-28 08:54:48,460 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:48,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:48,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:48,465 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:48,465 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,465 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:48,465 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:48,465 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,465 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,466 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,466 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:48,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:48,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:48,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:48,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:48,468 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:48,468 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:48,468 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:48,469 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:48,469 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:48,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:48,469 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 08:54:48,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:48,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:48,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:48,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:48,470 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:48,470 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:48,471 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#248 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:48,476 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#249 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:48,492 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/9e2b71357879416ba2f55301fe85190f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9e2b71357879416ba2f55301fe85190f 2023-04-28 08:54:48,497 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into 9e2b71357879416ba2f55301fe85190f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:48,497 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:48,497 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=11, startTime=1682672088454; duration=0sec 2023-04-28 08:54:48,497 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:48,497 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:48,498 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60840 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:48,499 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:48,499 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:48,499 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e50e3494117b4334b13bc6d16adb6e71, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a44794e5caa1441f873a6f32434a4d42, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/25eca1c028694effa820a2d7c831eab5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7d387fa114b84ca3bcbbddb8542ef6b0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/1f5ee6fe11ea4ea6945674d0b42d2980] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=59.4 K 2023-04-28 08:54:48,499 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e50e3494117b4334b13bc6d16adb6e71, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1682672085255 2023-04-28 08:54:48,500 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a44794e5caa1441f873a6f32434a4d42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672085411 2023-04-28 08:54:48,500 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 25eca1c028694effa820a2d7c831eab5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1682672086040 2023-04-28 08:54:48,500 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7d387fa114b84ca3bcbbddb8542ef6b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672086662 2023-04-28 08:54:48,500 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1f5ee6fe11ea4ea6945674d0b42d2980, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682672087289 2023-04-28 08:54:48,508 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#251 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:48,515 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/263485668dd5499fa531f13a2da8c631 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/263485668dd5499fa531f13a2da8c631 2023-04-28 08:54:48,519 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into 263485668dd5499fa531f13a2da8c631(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:48,519 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:48,519 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=11, startTime=1682672088455; duration=0sec 2023-04-28 08:54:48,519 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:48,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:48,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672148752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672148753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672148754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672148799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672148808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672148855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672148856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:48,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672148857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:48,892 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/84e7f9d6bfec488daec19a73521c8df1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/84e7f9d6bfec488daec19a73521c8df1 2023-04-28 08:54:48,893 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/056cddbc74ae40c187c5ae9bec8cc27a 2023-04-28 08:54:48,897 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into 84e7f9d6bfec488daec19a73521c8df1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:48,897 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:48,897 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=11, startTime=1682672088454; duration=0sec 2023-04-28 08:54:48,898 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:48,903 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/428b72946e5b4de9b3c309b05b49faeb 2023-04-28 08:54:49,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672149058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672149058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672149059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,313 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/006df79f32c049aaa7be51f2965c7f67 2023-04-28 08:54:49,317 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/056cddbc74ae40c187c5ae9bec8cc27a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/056cddbc74ae40c187c5ae9bec8cc27a 2023-04-28 08:54:49,321 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/056cddbc74ae40c187c5ae9bec8cc27a, entries=150, sequenceid=297, filesize=11.9 K 2023-04-28 08:54:49,322 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/428b72946e5b4de9b3c309b05b49faeb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/428b72946e5b4de9b3c309b05b49faeb 2023-04-28 08:54:49,325 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/428b72946e5b4de9b3c309b05b49faeb, entries=150, sequenceid=297, filesize=11.9 K 2023-04-28 08:54:49,326 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/006df79f32c049aaa7be51f2965c7f67 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/006df79f32c049aaa7be51f2965c7f67 2023-04-28 08:54:49,331 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/006df79f32c049aaa7be51f2965c7f67, entries=150, sequenceid=297, filesize=11.9 K 2023-04-28 08:54:49,331 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4def4a766691cce8b483578b280ed61b in 862ms, sequenceid=297, compaction requested=false 2023-04-28 08:54:49,331 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:49,332 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:49,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:49,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:49,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:49,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:49,336 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:49,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:49,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:49,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:49,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f97c921[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:49,338 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:49,338 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,340 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,340 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:49,340 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:49,340 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:49,340 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:49,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:49,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:49,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:49,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:49,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:49,344 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:49,344 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:49,344 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:49,345 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:49,345 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:49,346 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,346 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:49,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,346 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:49,346 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:49,348 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:49,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:49,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:49,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:49,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:49,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:49,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,350 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:49,350 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:49,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:49,364 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 08:54:49,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:49,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:49,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:49,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:49,364 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:49,364 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:49,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/e88380f79cbc4660ae38dfae8f51f135 2023-04-28 08:54:49,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672149382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672149382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672149383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/d498a459c46b41fb954a7eae3cef674c 2023-04-28 08:54:49,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/6d6f46d3089147128ad219ce39bd9e29 2023-04-28 08:54:49,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/e88380f79cbc4660ae38dfae8f51f135 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e88380f79cbc4660ae38dfae8f51f135 2023-04-28 08:54:49,407 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e88380f79cbc4660ae38dfae8f51f135, entries=200, sequenceid=314, filesize=14.3 K 2023-04-28 08:54:49,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/d498a459c46b41fb954a7eae3cef674c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/d498a459c46b41fb954a7eae3cef674c 2023-04-28 08:54:49,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/d498a459c46b41fb954a7eae3cef674c, entries=150, sequenceid=314, filesize=11.9 K 2023-04-28 08:54:49,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/6d6f46d3089147128ad219ce39bd9e29 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6d6f46d3089147128ad219ce39bd9e29 2023-04-28 08:54:49,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6d6f46d3089147128ad219ce39bd9e29, entries=150, sequenceid=314, filesize=11.9 K 2023-04-28 08:54:49,416 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4def4a766691cce8b483578b280ed61b in 52ms, sequenceid=314, compaction requested=true 2023-04-28 08:54:49,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:49,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:49,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:49,416 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:49,416 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:49,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:49,417 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:49,417 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:49,417 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:49,417 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:49,417 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:49,417 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:49,417 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/84e7f9d6bfec488daec19a73521c8df1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/428b72946e5b4de9b3c309b05b49faeb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/d498a459c46b41fb954a7eae3cef674c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=36.3 K 2023-04-28 08:54:49,417 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9e2b71357879416ba2f55301fe85190f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/056cddbc74ae40c187c5ae9bec8cc27a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e88380f79cbc4660ae38dfae8f51f135] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=38.7 K 2023-04-28 08:54:49,418 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 84e7f9d6bfec488daec19a73521c8df1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682672087289 2023-04-28 08:54:49,418 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9e2b71357879416ba2f55301fe85190f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682672087289 2023-04-28 08:54:49,418 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 428b72946e5b4de9b3c309b05b49faeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682672087619 2023-04-28 08:54:49,418 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 056cddbc74ae40c187c5ae9bec8cc27a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682672087619 2023-04-28 08:54:49,419 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d498a459c46b41fb954a7eae3cef674c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682672088748 2023-04-28 08:54:49,419 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e88380f79cbc4660ae38dfae8f51f135, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682672088748 2023-04-28 08:54:49,425 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:49,425 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:49,450 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:49,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:49,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:49,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:49,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:49,453 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:49,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:49,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:49,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:49,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:49,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:49,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:49,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:49,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:49,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:49,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:49,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:49,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:49,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:49,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:49,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:49,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:49,458 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:49,458 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:49,458 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 08:54:49,458 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:49,459 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:49,459 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:49,459 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:49,459 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:49,459 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:49,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:49,487 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:49,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672149496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672149496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672149498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672149600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672149600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672149601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672149803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672149803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:49,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672149804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:49,834 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/69c519cd264c4be4905efc581f677f07 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/69c519cd264c4be4905efc581f677f07 2023-04-28 08:54:49,841 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into 69c519cd264c4be4905efc581f677f07(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:49,841 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:49,841 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=13, startTime=1682672089416; duration=0sec 2023-04-28 08:54:49,841 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:49,841 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:49,842 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:49,842 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:49,842 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:49,843 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/263485668dd5499fa531f13a2da8c631, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/006df79f32c049aaa7be51f2965c7f67, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6d6f46d3089147128ad219ce39bd9e29] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=36.3 K 2023-04-28 08:54:49,844 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/8506bcc1e51b4876af38ab865eb4570c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8506bcc1e51b4876af38ab865eb4570c 2023-04-28 08:54:49,844 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 263485668dd5499fa531f13a2da8c631, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1682672087289 2023-04-28 08:54:49,844 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 006df79f32c049aaa7be51f2965c7f67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1682672087619 2023-04-28 08:54:49,845 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 6d6f46d3089147128ad219ce39bd9e29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682672088748 2023-04-28 08:54:49,849 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into 8506bcc1e51b4876af38ab865eb4570c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:49,849 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:49,849 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=13, startTime=1682672089416; duration=0sec 2023-04-28 08:54:49,849 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:49,855 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#260 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:49,864 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/4e25a6797608407db62f483bed79d1f4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4e25a6797608407db62f483bed79d1f4 2023-04-28 08:54:49,868 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into 4e25a6797608407db62f483bed79d1f4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:49,868 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:49,868 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=13, startTime=1682672089416; duration=0sec 2023-04-28 08:54:49,868 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/9093f52181d54e1ab51ecfe5fe1a9b5a 2023-04-28 08:54:49,868 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:50,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672150106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672150106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672150107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,280 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/a973797e5dce4aceb4920540f63640b7 2023-04-28 08:54:50,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672150610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672150612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672150613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,690 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/8f245813cf4b44cba4e86efbebd21cb6 2023-04-28 08:54:50,695 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/9093f52181d54e1ab51ecfe5fe1a9b5a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9093f52181d54e1ab51ecfe5fe1a9b5a 2023-04-28 08:54:50,699 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9093f52181d54e1ab51ecfe5fe1a9b5a, entries=150, sequenceid=332, filesize=11.9 K 2023-04-28 08:54:50,700 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/a973797e5dce4aceb4920540f63640b7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/a973797e5dce4aceb4920540f63640b7 2023-04-28 08:54:50,704 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/a973797e5dce4aceb4920540f63640b7, entries=150, sequenceid=332, filesize=11.9 K 2023-04-28 08:54:50,705 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/8f245813cf4b44cba4e86efbebd21cb6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8f245813cf4b44cba4e86efbebd21cb6 2023-04-28 08:54:50,710 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8f245813cf4b44cba4e86efbebd21cb6, entries=150, sequenceid=332, filesize=11.9 K 2023-04-28 08:54:50,711 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 4def4a766691cce8b483578b280ed61b in 1252ms, sequenceid=332, compaction requested=false 2023-04-28 08:54:50,711 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:50,711 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:50,711 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:50,711 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:50,711 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:50,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:50,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,716 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,716 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,716 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b50ca1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:50,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:50,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,719 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:50,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:50,719 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,719 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:50,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:50,720 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:50,721 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,721 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:50,723 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:50,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:50,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:50,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:50,723 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:50,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:50,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:50,729 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,730 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,732 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:50,732 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:50,732 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:50,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:50,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:50,732 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:50,732 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,733 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:50,733 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:50,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:50,733 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:50,733 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,733 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,733 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,733 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:50,807 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 08:54:50,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:50,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:50,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:50,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:50,807 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:50,807 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:50,833 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:50,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:50,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47150 deadline: 1682672150832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:50,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:50,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:50,836 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:50,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:50,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:50,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:50,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672150835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,838 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:50,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:50,838 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,839 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,839 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:50,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:50,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:50,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:50,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:50,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:50,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:50,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:50,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:50,842 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:50,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@710a7d20[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:50,843 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:50,843 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,845 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,846 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:50,846 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:50,846 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:50,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:50,847 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:50,847 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:50,847 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,847 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:50,850 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:50,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:50,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:50,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:50,850 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:50,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,851 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:50,852 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:50,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:50,856 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:50,856 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672150936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:50,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672150938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,956 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:50,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:50,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:50,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:50,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:50,960 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:50,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:50,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:50,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:50,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,962 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:50,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:50,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:50,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:50,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:50,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:50,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:50,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:50,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:50,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:50,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:50,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:50,964 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:50,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:50,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:50,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:50,964 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:50,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,965 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:50,967 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:50,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1d6d22[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:50,967 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:50,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:50,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:50,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:50,973 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,973 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:50,973 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,973 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:50,976 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:50,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:50,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:50,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:50,976 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:50,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,977 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:50,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:50,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:50,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:50,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:50,981 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:50,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:50,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:50,981 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:50,982 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:50,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,082 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:51,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:51,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:51,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:51,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:51,085 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:51,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:51,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:51,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:51,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:51,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:51,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:51,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:51,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:51,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:51,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:51,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:51,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:51,088 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:51,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:51,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:51,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:51,091 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,091 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,091 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,091 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:51,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:51,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df27d41[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:51,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:51,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:51,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:51,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:51,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:51,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:51,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:51,097 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:51,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:51,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:51,097 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:51,097 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:51,099 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,099 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,099 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,099 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,100 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:51,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:51,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:51,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,104 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,104 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:51,104 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,105 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,105 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672151137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672151141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,205 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:51,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:51,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:51,208 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:51,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:51,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:51,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,209 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:51,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:51,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:51,211 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,211 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:51,211 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:51,211 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:51,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:51,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,212 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,213 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,213 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:51,214 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:51,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5639a05a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:51,214 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/78ec2cdb55a44d34b93797e4b1b81b14 2023-04-28 08:54:51,215 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:51,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:51,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:51,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,216 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:51,218 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:51,218 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,218 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:51,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:51,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:51,221 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:51,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:51,222 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:51,222 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:51,222 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:51,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,225 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/40a0e7ced4fe4aa9a697710db1017dd1 2023-04-28 08:54:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,226 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:51,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,235 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,235 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,236 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,236 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:51,236 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:51,336 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:51,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:51,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:51,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:51,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:51,339 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:51,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:51,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:51,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,340 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,340 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:51,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:51,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:51,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:51,342 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:51,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:51,343 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:51,347 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,348 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,348 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,348 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:51,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f834c0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:51,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:51,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,350 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:51,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:51,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:51,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:51,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,352 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,352 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,352 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:51,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:51,354 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:51,355 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:51,355 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:51,355 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:51,355 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:51,362 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,362 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,362 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,363 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,364 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:51,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,368 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:51,368 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:51,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672151442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672151445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,468 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:51,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:51,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:51,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:51,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:51,471 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:51,471 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,471 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:51,471 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:51,472 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,472 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,473 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,473 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,473 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,473 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:51,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:51,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:51,475 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:51,475 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,475 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:51,475 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:51,475 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:51,475 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,476 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:51,476 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:51,476 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:51,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:51,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:51,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:51,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,477 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,478 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,478 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:51,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@631b3300[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:51,479 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:51,479 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:51,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:51,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:51,482 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:51,483 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:51,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:51,486 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:51,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:51,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:51,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:51,486 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:51,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,487 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,491 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,491 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,491 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,491 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:51,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:51,491 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,492 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,492 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:51,492 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,492 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:51,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,492 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,492 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:51,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,593 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:51,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:51,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:51,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:51,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:51,596 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:51,596 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,596 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:51,596 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:51,596 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,596 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,599 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:51,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:51,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:51,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:51,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:51,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:51,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:51,601 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:51,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:51,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:51,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:51,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:51,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:51,603 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,603 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,603 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,604 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:51,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ddd4257[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:51,605 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:51,605 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:51,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:51,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:51,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:51,607 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,607 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:51,608 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,608 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:51,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:51,610 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:51,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:51,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:51,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:51,610 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:51,613 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,613 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,613 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:51,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,615 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:51,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:51,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,617 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:51,618 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,617 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:51,618 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:51,618 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:51,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:51,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:51,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47086 deadline: 1682672151618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47088 deadline: 1682672151619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47074 deadline: 1682672151623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/4ded6a5c5e774cef8151a5196f3290ca 2023-04-28 08:54:51,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/78ec2cdb55a44d34b93797e4b1b81b14 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/78ec2cdb55a44d34b93797e4b1b81b14 2023-04-28 08:54:51,654 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/78ec2cdb55a44d34b93797e4b1b81b14, entries=200, sequenceid=350, filesize=14.3 K 2023-04-28 08:54:51,655 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/40a0e7ced4fe4aa9a697710db1017dd1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/40a0e7ced4fe4aa9a697710db1017dd1 2023-04-28 08:54:51,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/40a0e7ced4fe4aa9a697710db1017dd1, entries=150, sequenceid=350, filesize=11.9 K 2023-04-28 08:54:51,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/4ded6a5c5e774cef8151a5196f3290ca as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4ded6a5c5e774cef8151a5196f3290ca 2023-04-28 08:54:51,662 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4ded6a5c5e774cef8151a5196f3290ca, entries=150, sequenceid=350, filesize=11.9 K 2023-04-28 08:54:51,663 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4def4a766691cce8b483578b280ed61b in 856ms, sequenceid=350, compaction requested=true 2023-04-28 08:54:51,663 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:51,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:51,663 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:51,664 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:51,663 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:51,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:51,664 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:51,665 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:51,665 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:51,665 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:51,665 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,665 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,665 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/69c519cd264c4be4905efc581f677f07, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9093f52181d54e1ab51ecfe5fe1a9b5a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/78ec2cdb55a44d34b93797e4b1b81b14] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=38.8 K 2023-04-28 08:54:51,665 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8506bcc1e51b4876af38ab865eb4570c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/a973797e5dce4aceb4920540f63640b7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/40a0e7ced4fe4aa9a697710db1017dd1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=36.4 K 2023-04-28 08:54:51,665 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 69c519cd264c4be4905efc581f677f07, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682672088748 2023-04-28 08:54:51,665 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8506bcc1e51b4876af38ab865eb4570c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682672088748 2023-04-28 08:54:51,666 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9093f52181d54e1ab51ecfe5fe1a9b5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682672089375 2023-04-28 08:54:51,666 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a973797e5dce4aceb4920540f63640b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682672089375 2023-04-28 08:54:51,666 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 78ec2cdb55a44d34b93797e4b1b81b14, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682672089495 2023-04-28 08:54:51,666 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 40a0e7ced4fe4aa9a697710db1017dd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682672089495 2023-04-28 08:54:51,674 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:51,674 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:51,731 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:51,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:51,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:51,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:51,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:51,734 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:51,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,735 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:51,735 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:51,735 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,735 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:51,736 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,736 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,737 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:51,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:51,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:51,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:51,737 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:51,737 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:51,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:51,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:51,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:51,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:51,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:51,738 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 08:54:51,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:51,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:51,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:51,738 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:51,739 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:51,739 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:51,751 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/348c52da61804a74997d633daafb91cc 2023-04-28 08:54:51,764 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/852f33b6a1d5431c95d8341065102b4e 2023-04-28 08:54:51,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:51,950 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:51,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672151969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:51,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:51,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672151971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672152072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672152075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,083 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/3426b9031d4c42f0acb811818ed0266a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/3426b9031d4c42f0acb811818ed0266a 2023-04-28 08:54:52,083 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/3b60722361854780beeb6260611377dd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3b60722361854780beeb6260611377dd 2023-04-28 08:54:52,091 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into 3b60722361854780beeb6260611377dd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:52,092 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:52,092 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=13, startTime=1682672091663; duration=0sec 2023-04-28 08:54:52,092 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:52,092 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:52,093 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:52,093 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into 3426b9031d4c42f0acb811818ed0266a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:52,093 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:52,093 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:52,093 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=13, startTime=1682672091663; duration=0sec 2023-04-28 08:54:52,093 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,093 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:52,093 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4e25a6797608407db62f483bed79d1f4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8f245813cf4b44cba4e86efbebd21cb6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4ded6a5c5e774cef8151a5196f3290ca] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=36.4 K 2023-04-28 08:54:52,093 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4e25a6797608407db62f483bed79d1f4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1682672088748 2023-04-28 08:54:52,094 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8f245813cf4b44cba4e86efbebd21cb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1682672089375 2023-04-28 08:54:52,094 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4ded6a5c5e774cef8151a5196f3290ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682672089495 2023-04-28 08:54:52,100 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:52,109 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/67ea3187a3004023a86d40d2bb36503f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/67ea3187a3004023a86d40d2bb36503f 2023-04-28 08:54:52,114 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into 67ea3187a3004023a86d40d2bb36503f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:52,114 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:52,114 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=13, startTime=1682672091664; duration=0sec 2023-04-28 08:54:52,114 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:52,174 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/dd5cbf22617847758b84e511348b252f 2023-04-28 08:54:52,179 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/348c52da61804a74997d633daafb91cc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/348c52da61804a74997d633daafb91cc 2023-04-28 08:54:52,183 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/348c52da61804a74997d633daafb91cc, entries=150, sequenceid=367, filesize=11.9 K 2023-04-28 08:54:52,183 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/852f33b6a1d5431c95d8341065102b4e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/852f33b6a1d5431c95d8341065102b4e 2023-04-28 08:54:52,187 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/852f33b6a1d5431c95d8341065102b4e, entries=150, sequenceid=367, filesize=11.9 K 2023-04-28 08:54:52,188 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/dd5cbf22617847758b84e511348b252f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/dd5cbf22617847758b84e511348b252f 2023-04-28 08:54:52,191 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/dd5cbf22617847758b84e511348b252f, entries=150, sequenceid=367, filesize=11.9 K 2023-04-28 08:54:52,192 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4def4a766691cce8b483578b280ed61b in 454ms, sequenceid=367, compaction requested=false 2023-04-28 08:54:52,192 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:52,192 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,192 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:52,192 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:52,192 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,192 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:52,192 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:52,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267d0ec6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:52,196 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:52,196 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,198 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,198 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:52,198 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:52,198 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:52,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:52,199 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:52,200 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,200 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:52,199 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:52,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:52,202 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:52,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:52,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:52,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:52,202 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:52,204 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,204 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,204 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,204 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:52,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:52,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,211 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:52,211 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:52,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:52,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:52,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:52,278 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:52,297 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672152295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672152297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,311 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:52,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:52,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:52,315 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:52,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:52,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:52,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:52,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:52,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:52,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:52,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:52,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:52,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:52,319 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,319 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:52,319 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:52,319 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:52,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:52,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:52,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:52,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@139670fd[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:52,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:52,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,323 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,323 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:52,323 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:52,323 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:52,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:52,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:52,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:52,328 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:52,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:52,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:52,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:52,328 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:52,330 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,330 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,330 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,330 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,333 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,333 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,333 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:52,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:52,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:52,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,334 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:52,334 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:52,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672152398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672152400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,434 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:52,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:52,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:52,437 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:52,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:52,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:52,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,439 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:52,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:52,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:52,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:52,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:52,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:52,441 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,441 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:52,441 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:52,441 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:52,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:52,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:52,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:52,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292fc2bc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:52,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:52,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:52,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:52,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:52,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:52,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:52,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:52,451 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:52,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:52,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:52,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:52,452 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:52,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,455 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:52,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:52,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,459 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:52,459 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,459 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:52,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,459 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:52,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,559 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:52,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:52,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:52,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:52,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:52,562 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:52,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:52,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:52,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,564 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:52,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:52,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:52,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:52,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:52,566 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:52,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:52,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:52,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e37f234[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:52,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,571 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:52,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:52,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:52,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:52,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:52,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:52,576 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:52,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:52,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:52,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:52,576 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:52,577 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,577 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,578 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:52,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:52,581 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,582 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,582 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,582 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:52,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672152601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672152604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,682 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:52,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:52,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:52,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:52,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:52,685 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:52,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:52,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:52,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,691 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/910fd916873a46e3bd2639ead0721b1e 2023-04-28 08:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:52,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:52,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:52,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:52,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:52,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:52,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:52,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:52,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:52,694 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:52,694 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,694 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:52,694 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:52,695 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,695 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:52,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3072d80a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:52,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:52,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,701 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,701 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:52,701 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:52,701 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:52,702 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,702 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:52,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,703 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:52,703 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,703 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:52,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:52,706 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:52,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:52,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:52,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:52,707 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:52,709 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,709 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,709 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,714 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:52,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:52,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,715 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:52,715 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:52,815 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:52,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:52,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:52,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:52,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:52,820 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:52,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:52,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:52,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,822 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,822 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,822 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:52,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:52,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:52,824 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,824 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:52,824 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:52,824 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:52,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:52,826 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,826 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,826 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,826 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:52,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10c34761[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:52,828 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:52,828 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:52,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:52,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:52,831 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:52,832 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,832 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:52,832 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,832 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:52,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:52,835 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:52,835 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:52,835 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:52,835 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:52,835 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:52,836 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,836 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,836 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,837 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,838 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,840 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,840 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,840 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,840 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,840 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,840 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:52,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:52,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:52,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,841 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:52,841 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:52,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672152904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672152910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,941 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:52,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:52,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:52,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:52,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:52,944 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:52,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:52,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:52,945 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,945 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,947 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,947 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:52,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:52,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:52,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:52,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:52,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:52,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:52,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:52,949 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,949 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:52,949 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:52,950 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:52,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:52,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:52,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:52,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:52,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,951 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:52,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47ae5de7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:52,953 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:52,953 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,954 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,954 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:52,954 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:52,954 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:52,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:52,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,955 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:52,955 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,955 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:52,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:52,959 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:52,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:52,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:52,959 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:52,959 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:52,961 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,961 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:52,962 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:52,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:52,965 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:52,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:52,966 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:52,966 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:52,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:52,966 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,965 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:52,966 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:52,966 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:52,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:52,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:52,967 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:52,967 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:52,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,067 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:53,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:53,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:53,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:53,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:53,070 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:53,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:53,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:53,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,072 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:53,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:53,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:53,074 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,074 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:53,074 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:53,074 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:53,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:53,079 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@649a174[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:53,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:53,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,082 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:53,083 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,083 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:53,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:53,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:53,084 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,084 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:53,084 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,084 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:53,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:53,087 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:53,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:53,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:53,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:53,087 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:53,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,089 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,090 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:53,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:53,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,095 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:53,095 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:53,117 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/6ef3589cb15e49718fc489c91ac2c9c8 2023-04-28 08:54:53,195 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:53,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:53,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:53,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:53,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:53,198 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:53,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:53,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:53,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,200 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,200 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,200 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,200 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:53,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:53,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:53,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:53,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:53,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:53,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:53,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:53,202 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,202 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:53,202 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:53,202 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:53,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:53,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:53,202 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:53,203 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,204 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:53,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20dd9b3f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:53,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:53,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:53,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:53,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:53,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:53,210 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,210 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,210 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,210 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:53,212 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:53,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:53,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:53,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:53,212 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:53,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,216 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:53,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:53,218 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,219 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,219 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:53,219 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,219 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:53,219 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,219 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,319 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:53,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:53,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:53,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:53,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:53,322 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:53,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:53,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:53,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:53,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:53,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:53,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:53,326 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,326 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:53,326 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:53,326 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:53,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:53,329 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,329 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,330 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,330 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:53,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a2f280a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:53,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:53,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:53,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:53,332 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:53,332 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,333 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:53,334 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:53,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,334 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,334 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:53,336 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:53,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:53,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:53,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:53,336 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:53,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,338 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,342 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:53,343 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:53,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:53,343 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,343 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,343 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,343 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,343 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:53,343 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672153410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672153414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,443 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:53,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:53,446 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:53,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:53,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:53,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:53,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:53,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:53,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:53,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:53,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:53,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:53,449 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,449 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:53,449 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:53,449 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,450 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:53,450 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:53,450 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,450 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:53,450 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:53,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:53,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5b09cd[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:53,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:53,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:53,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:53,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:53,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:53,459 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:53,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:53,461 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:53,461 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:53,461 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:53,461 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:53,461 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:53,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:53,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,464 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:53,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:53,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:53,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:53,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:53,468 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:53,468 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:53,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/016db0d9722d453a8c37d3396e49ea26 2023-04-28 08:54:53,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/910fd916873a46e3bd2639ead0721b1e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/910fd916873a46e3bd2639ead0721b1e 2023-04-28 08:54:53,534 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/910fd916873a46e3bd2639ead0721b1e, entries=150, sequenceid=386, filesize=11.9 K 2023-04-28 08:54:53,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/6ef3589cb15e49718fc489c91ac2c9c8 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6ef3589cb15e49718fc489c91ac2c9c8 2023-04-28 08:54:53,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6ef3589cb15e49718fc489c91ac2c9c8, entries=150, sequenceid=386, filesize=11.9 K 2023-04-28 08:54:53,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/016db0d9722d453a8c37d3396e49ea26 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/016db0d9722d453a8c37d3396e49ea26 2023-04-28 08:54:53,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/016db0d9722d453a8c37d3396e49ea26, entries=150, sequenceid=386, filesize=11.9 K 2023-04-28 08:54:53,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 4def4a766691cce8b483578b280ed61b in 1272ms, sequenceid=386, compaction requested=true 2023-04-28 08:54:53,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:53,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:53,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:53,549 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:53,549 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:53,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:53,551 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:53,551 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:53,551 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,551 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3b60722361854780beeb6260611377dd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/348c52da61804a74997d633daafb91cc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/910fd916873a46e3bd2639ead0721b1e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=36.5 K 2023-04-28 08:54:53,551 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:53,551 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:53,551 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,551 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/3426b9031d4c42f0acb811818ed0266a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/852f33b6a1d5431c95d8341065102b4e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6ef3589cb15e49718fc489c91ac2c9c8] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=36.5 K 2023-04-28 08:54:53,551 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 3b60722361854780beeb6260611377dd, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682672089495 2023-04-28 08:54:53,552 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3426b9031d4c42f0acb811818ed0266a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682672089495 2023-04-28 08:54:53,555 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 348c52da61804a74997d633daafb91cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682672090824 2023-04-28 08:54:53,555 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 852f33b6a1d5431c95d8341065102b4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682672090824 2023-04-28 08:54:53,555 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 910fd916873a46e3bd2639ead0721b1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682672091968 2023-04-28 08:54:53,555 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6ef3589cb15e49718fc489c91ac2c9c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682672091968 2023-04-28 08:54:53,563 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:53,563 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:53,568 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:53,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:53,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:53,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:53,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:53,577 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:53,577 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,577 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:53,577 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:53,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:53,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:53,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,580 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:53,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:53,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:53,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:53,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:53,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:53,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:53,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:53,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:53,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:53,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:53,590 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:54:53,590 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:53,591 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:53,591 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:53,591 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:53,591 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:53,591 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:53,596 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/81e204af1c7b4a9387f81f330bdfb6de as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/81e204af1c7b4a9387f81f330bdfb6de 2023-04-28 08:54:53,599 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/6aa9f91ee6474764ac521a7c47a52357 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/6aa9f91ee6474764ac521a7c47a52357 2023-04-28 08:54:53,602 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into 81e204af1c7b4a9387f81f330bdfb6de(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:53,602 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:53,602 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=13, startTime=1682672093549; duration=0sec 2023-04-28 08:54:53,602 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:53,602 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:54:53,603 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:54:53,603 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:53,603 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:53,604 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/67ea3187a3004023a86d40d2bb36503f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/dd5cbf22617847758b84e511348b252f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/016db0d9722d453a8c37d3396e49ea26] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=36.5 K 2023-04-28 08:54:53,604 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 67ea3187a3004023a86d40d2bb36503f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1682672089495 2023-04-28 08:54:53,605 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dd5cbf22617847758b84e511348b252f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1682672090824 2023-04-28 08:54:53,605 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 016db0d9722d453a8c37d3396e49ea26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682672091968 2023-04-28 08:54:53,605 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into 6aa9f91ee6474764ac521a7c47a52357(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:53,605 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:53,605 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=13, startTime=1682672093549; duration=0sec 2023-04-28 08:54:53,605 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:53,615 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/59b8b8215c2e43a590e8fe044c56d959 2023-04-28 08:54:53,620 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:53,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:53,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:53,627 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/7473fa8635c944b69fbfb0be51be5e15 2023-04-28 08:54:53,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672153642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672153643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672153644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672153744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672153746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672153746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672153945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,949 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672153948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:53,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672153950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,031 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/2040a2c854b747ab8d2ca243a8aa170a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/2040a2c854b747ab8d2ca243a8aa170a 2023-04-28 08:54:54,036 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into 2040a2c854b747ab8d2ca243a8aa170a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:54,036 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:54,036 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=13, startTime=1682672093549; duration=0sec 2023-04-28 08:54:54,036 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:54,039 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/5b86907ea65e486b9d6c9365aebc0afd 2023-04-28 08:54:54,043 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/59b8b8215c2e43a590e8fe044c56d959 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/59b8b8215c2e43a590e8fe044c56d959 2023-04-28 08:54:54,047 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/59b8b8215c2e43a590e8fe044c56d959, entries=100, sequenceid=402, filesize=9.5 K 2023-04-28 08:54:54,047 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/7473fa8635c944b69fbfb0be51be5e15 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7473fa8635c944b69fbfb0be51be5e15 2023-04-28 08:54:54,051 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7473fa8635c944b69fbfb0be51be5e15, entries=100, sequenceid=402, filesize=9.5 K 2023-04-28 08:54:54,059 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/5b86907ea65e486b9d6c9365aebc0afd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5b86907ea65e486b9d6c9365aebc0afd 2023-04-28 08:54:54,063 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5b86907ea65e486b9d6c9365aebc0afd, entries=100, sequenceid=402, filesize=9.5 K 2023-04-28 08:54:54,064 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 4def4a766691cce8b483578b280ed61b in 474ms, sequenceid=402, compaction requested=false 2023-04-28 08:54:54,064 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:54,064 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:54,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:54,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:54,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:54,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:54,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:54,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:54,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54524953[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:54,068 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:54,068 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,069 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:54,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:54,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:54,069 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,069 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:54,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,070 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:54,071 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:54,071 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:54,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:54,073 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:54,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:54,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:54,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:54,073 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:54,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:54,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:54,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:54,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:54,077 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:54,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:54,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:54,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:54,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:54,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:54,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:54,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:54,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:54,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:54,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:54,081 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:54,081 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:54,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:54,181 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:54,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:54,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:54,184 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:54,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:54,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:54,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:54,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:54,185 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:54,186 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:54,186 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:54,186 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:54,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:54,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:54,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:54,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:54,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:54,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:54,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:54,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:54,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:54,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:54,187 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:54,187 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:54,187 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 08:54:54,188 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:54,188 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:54,188 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:54,188 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:54,188 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:54,188 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:54,252 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:54,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672154262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672154262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,265 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672154263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672154364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672154366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672154366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672154416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,424 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672154423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672154568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672154570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672154570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,595 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/4e8bbc105c974c589edcae01da0a9b71 2023-04-28 08:54:54,622 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/0e0c7f04477644feb4e6de13ab725a20 2023-04-28 08:54:54,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672154870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672154873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:54,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672154874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,032 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e7daac3149d3438a8d3e8488220e5b9f 2023-04-28 08:54:55,036 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/4e8bbc105c974c589edcae01da0a9b71 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4e8bbc105c974c589edcae01da0a9b71 2023-04-28 08:54:55,040 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4e8bbc105c974c589edcae01da0a9b71, entries=150, sequenceid=422, filesize=11.9 K 2023-04-28 08:54:55,041 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/0e0c7f04477644feb4e6de13ab725a20 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/0e0c7f04477644feb4e6de13ab725a20 2023-04-28 08:54:55,044 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/0e0c7f04477644feb4e6de13ab725a20, entries=150, sequenceid=422, filesize=11.9 K 2023-04-28 08:54:55,045 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e7daac3149d3438a8d3e8488220e5b9f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e7daac3149d3438a8d3e8488220e5b9f 2023-04-28 08:54:55,049 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e7daac3149d3438a8d3e8488220e5b9f, entries=150, sequenceid=422, filesize=11.9 K 2023-04-28 08:54:55,049 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 4def4a766691cce8b483578b280ed61b in 862ms, sequenceid=422, compaction requested=true 2023-04-28 08:54:55,050 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:55,050 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:55,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:55,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:55,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:55,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:55,052 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,052 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:55,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:55,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@189a2485[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:55,054 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:55,054 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,055 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:55,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:55,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:55,056 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,056 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:55,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:55,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:55,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:55,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:55,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:55,059 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:55,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:55,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:55,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:55,059 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:55,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:55,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:55,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:55,062 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:55,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:55,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:55,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:55,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:55,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,066 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,066 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:55,066 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:55,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:55,167 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:55,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:55,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:55,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:55,170 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:55,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:55,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:55,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:55,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:55,171 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:55,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:55,172 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,172 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:55,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:55,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:55,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:55,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:55,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:55,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:55,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:55,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:55,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:55,173 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:55,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:55,174 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-28 08:54:55,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:55,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:55,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:55,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:55,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:55,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:55,379 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:55,401 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672155399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672155399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672155399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672155502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672155502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672155503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,585 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/8d1c7daf0d074c65953541890bc899eb 2023-04-28 08:54:55,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672155705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672155706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672155706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:55,994 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/c0f74ae88b4a4c2fb8985f1936b9b32e 2023-04-28 08:54:56,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672156008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672156010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672156011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,404 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=437 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e0528f29f7344054978e8e30958bbaa2 2023-04-28 08:54:56,408 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/8d1c7daf0d074c65953541890bc899eb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/8d1c7daf0d074c65953541890bc899eb 2023-04-28 08:54:56,412 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/8d1c7daf0d074c65953541890bc899eb, entries=150, sequenceid=437, filesize=11.9 K 2023-04-28 08:54:56,413 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/c0f74ae88b4a4c2fb8985f1936b9b32e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/c0f74ae88b4a4c2fb8985f1936b9b32e 2023-04-28 08:54:56,417 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/c0f74ae88b4a4c2fb8985f1936b9b32e, entries=150, sequenceid=437, filesize=11.9 K 2023-04-28 08:54:56,417 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e0528f29f7344054978e8e30958bbaa2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e0528f29f7344054978e8e30958bbaa2 2023-04-28 08:54:56,421 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e0528f29f7344054978e8e30958bbaa2, entries=150, sequenceid=437, filesize=11.9 K 2023-04-28 08:54:56,422 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4def4a766691cce8b483578b280ed61b in 1248ms, sequenceid=437, compaction requested=true 2023-04-28 08:54:56,422 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:56,422 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:56,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:56,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:56,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:56,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,423 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,424 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,424 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:56,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ba620e4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:56,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:56,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:56,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:56,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:56,427 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:56,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:56,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 08:54:56,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:56,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:56,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:56,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:56,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:56,429 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:56,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:56,433 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:56,433 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:56,433 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:56,433 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:56,433 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:56,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,439 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:56,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:56,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,442 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,442 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:56,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672156456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672156457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672156515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672156517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672156518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,542 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:56,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:56,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:56,549 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:56,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:56,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:56,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,551 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,551 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,552 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,552 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:56,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:56,553 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:56,553 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,553 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:56,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:56,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:56,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:56,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:56,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:56,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:56,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:56,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:56,559 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c744d3c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:56,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:56,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,562 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672156560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:56,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672156560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,562 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:56,562 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:56,562 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:56,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:56,563 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:56,566 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:56,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:56,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:56,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:56,566 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:56,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,572 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:56,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:56,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,573 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:56,573 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,673 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:56,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:56,677 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:56,677 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,677 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:56,677 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:56,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,679 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:56,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:56,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:56,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:56,680 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:56,680 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,680 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:56,681 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:56,681 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:56,681 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,681 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:56,681 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:56,681 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,681 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:56,681 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:56,681 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,681 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:56,681 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:56,683 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:56,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aaa8530[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:56,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:56,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:56,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:56,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:56,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:56,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:56,688 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:56,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:56,690 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:56,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:56,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:56,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:56,690 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:56,692 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,692 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,692 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,698 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:56,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:56,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:56,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,699 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:56,699 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:56,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672156763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:56,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672156763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,799 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:56,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:56,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:56,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:56,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:56,802 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:56,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:56,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:56,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,806 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:56,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:56,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:56,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:56,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:56,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:56,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:56,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:56,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:56,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:56,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:56,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:56,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:56,809 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,809 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,809 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,809 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:56,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35d2eec[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:56,811 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:56,811 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,812 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,812 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:56,812 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:56,812 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:56,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:56,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:56,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:56,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:56,814 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:56,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:56,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:56,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:56,814 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:56,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:56,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:56,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,817 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,818 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:56,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:56,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:54:56,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:56,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:56,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:56,824 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:56,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:56,824 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:56,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/f0f6533cb7424c6faa14ffe89236e159 2023-04-28 08:54:56,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/38e8301f4883440ead50da13bc2cc46a 2023-04-28 08:54:56,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/4429f107ae3f4b15b02c0022fbfde633 2023-04-28 08:54:56,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/f0f6533cb7424c6faa14ffe89236e159 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/f0f6533cb7424c6faa14ffe89236e159 2023-04-28 08:54:56,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/f0f6533cb7424c6faa14ffe89236e159, entries=200, sequenceid=455, filesize=14.3 K 2023-04-28 08:54:56,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/38e8301f4883440ead50da13bc2cc46a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/38e8301f4883440ead50da13bc2cc46a 2023-04-28 08:54:56,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/38e8301f4883440ead50da13bc2cc46a, entries=150, sequenceid=455, filesize=11.9 K 2023-04-28 08:54:56,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/4429f107ae3f4b15b02c0022fbfde633 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4429f107ae3f4b15b02c0022fbfde633 2023-04-28 08:54:56,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4429f107ae3f4b15b02c0022fbfde633, entries=150, sequenceid=455, filesize=11.9 K 2023-04-28 08:54:56,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 4def4a766691cce8b483578b280ed61b in 445ms, sequenceid=455, compaction requested=true 2023-04-28 08:54:56,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:56,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:56,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:56,874 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:56,874 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:56,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:56,875 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61812 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:56,875 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59372 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:56,875 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/A is initiating minor compaction (all files) 2023-04-28 08:54:56,875 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/B is initiating minor compaction (all files) 2023-04-28 08:54:56,875 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/A in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,875 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/6aa9f91ee6474764ac521a7c47a52357, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/59b8b8215c2e43a590e8fe044c56d959, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4e8bbc105c974c589edcae01da0a9b71, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/8d1c7daf0d074c65953541890bc899eb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/f0f6533cb7424c6faa14ffe89236e159] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=60.4 K 2023-04-28 08:54:56,876 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/B in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,876 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/81e204af1c7b4a9387f81f330bdfb6de, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7473fa8635c944b69fbfb0be51be5e15, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/0e0c7f04477644feb4e6de13ab725a20, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/c0f74ae88b4a4c2fb8985f1936b9b32e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/38e8301f4883440ead50da13bc2cc46a] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=58.0 K 2023-04-28 08:54:56,876 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 6aa9f91ee6474764ac521a7c47a52357, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682672091968 2023-04-28 08:54:56,876 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 81e204af1c7b4a9387f81f330bdfb6de, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682672091968 2023-04-28 08:54:56,876 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 59b8b8215c2e43a590e8fe044c56d959, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1682672092286 2023-04-28 08:54:56,876 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7473fa8635c944b69fbfb0be51be5e15, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1682672092286 2023-04-28 08:54:56,876 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4e8bbc105c974c589edcae01da0a9b71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1682672093626 2023-04-28 08:54:56,877 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0e0c7f04477644feb4e6de13ab725a20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1682672093626 2023-04-28 08:54:56,877 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8d1c7daf0d074c65953541890bc899eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1682672094258 2023-04-28 08:54:56,877 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c0f74ae88b4a4c2fb8985f1936b9b32e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1682672094258 2023-04-28 08:54:56,877 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f0f6533cb7424c6faa14ffe89236e159, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682672095398 2023-04-28 08:54:56,877 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 38e8301f4883440ead50da13bc2cc46a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682672095398 2023-04-28 08:54:56,884 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#B#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:56,885 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#A#compaction#290 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:56,924 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:56,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:56,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:56,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:56,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:56,928 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:56,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:56,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:56,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:56,929 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:56,930 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:56,930 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:56,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:56,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:56,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:56,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:56,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:56,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:56,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:56,931 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:56,931 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:56,931 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:56,931 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-28 08:54:56,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:56,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:56,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:56,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:56,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:56,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:57,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:57,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:57,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672157096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672157099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672157202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672157202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,294 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/bf9960257abf43feab43fe2c198e206a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/bf9960257abf43feab43fe2c198e206a 2023-04-28 08:54:57,298 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/B of 4def4a766691cce8b483578b280ed61b into bf9960257abf43feab43fe2c198e206a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:57,299 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:57,299 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/B, priority=11, startTime=1682672096874; duration=0sec 2023-04-28 08:54:57,299 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:54:57,299 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:54:57,301 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59372 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:54:57,301 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 4def4a766691cce8b483578b280ed61b/C is initiating minor compaction (all files) 2023-04-28 08:54:57,301 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4def4a766691cce8b483578b280ed61b/C in TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:57,301 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/2040a2c854b747ab8d2ca243a8aa170a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5b86907ea65e486b9d6c9365aebc0afd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e7daac3149d3438a8d3e8488220e5b9f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e0528f29f7344054978e8e30958bbaa2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4429f107ae3f4b15b02c0022fbfde633] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp, totalSize=58.0 K 2023-04-28 08:54:57,302 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2040a2c854b747ab8d2ca243a8aa170a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1682672091968 2023-04-28 08:54:57,302 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5b86907ea65e486b9d6c9365aebc0afd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1682672092286 2023-04-28 08:54:57,303 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e7daac3149d3438a8d3e8488220e5b9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1682672093626 2023-04-28 08:54:57,303 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e0528f29f7344054978e8e30958bbaa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=437, earliestPutTs=1682672094258 2023-04-28 08:54:57,304 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4429f107ae3f4b15b02c0022fbfde633, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1682672095398 2023-04-28 08:54:57,305 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/41ef422e1b334a0689e324268258806c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/41ef422e1b334a0689e324268258806c 2023-04-28 08:54:57,310 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/A of 4def4a766691cce8b483578b280ed61b into 41ef422e1b334a0689e324268258806c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:57,310 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:57,310 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/A, priority=11, startTime=1682672096874; duration=0sec 2023-04-28 08:54:57,310 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:57,314 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 4def4a766691cce8b483578b280ed61b#C#compaction#293 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:54:57,341 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7daa2d4b71e840c2a243fe4470e8c4d6 2023-04-28 08:54:57,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672157405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672157405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47074 deadline: 1682672157521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47088 deadline: 1682672157523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47086 deadline: 1682672157525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47108 deadline: 1682672157707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:54:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47150 deadline: 1682672157708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,726 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/e218ec7a92bd46bb881b9571f271aa55 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e218ec7a92bd46bb881b9571f271aa55 2023-04-28 08:54:57,731 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 4def4a766691cce8b483578b280ed61b/C of 4def4a766691cce8b483578b280ed61b into e218ec7a92bd46bb881b9571f271aa55(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:54:57,731 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:57,731 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b., storeName=4def4a766691cce8b483578b280ed61b/C, priority=11, startTime=1682672096874; duration=0sec 2023-04-28 08:54:57,731 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:54:57,750 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/b83abd1b9d30415b81319e88406e143e 2023-04-28 08:54:57,762 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/451ffbaee6ce4b06ae0dbd70c0874e30 2023-04-28 08:54:57,765 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7daa2d4b71e840c2a243fe4470e8c4d6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7daa2d4b71e840c2a243fe4470e8c4d6 2023-04-28 08:54:57,769 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7daa2d4b71e840c2a243fe4470e8c4d6, entries=150, sequenceid=469, filesize=11.9 K 2023-04-28 08:54:57,769 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/b83abd1b9d30415b81319e88406e143e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b83abd1b9d30415b81319e88406e143e 2023-04-28 08:54:57,773 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b83abd1b9d30415b81319e88406e143e, entries=150, sequenceid=469, filesize=11.9 K 2023-04-28 08:54:57,773 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/451ffbaee6ce4b06ae0dbd70c0874e30 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/451ffbaee6ce4b06ae0dbd70c0874e30 2023-04-28 08:54:57,778 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/451ffbaee6ce4b06ae0dbd70c0874e30, entries=150, sequenceid=469, filesize=11.9 K 2023-04-28 08:54:57,779 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 4def4a766691cce8b483578b280ed61b in 848ms, sequenceid=469, compaction requested=false 2023-04-28 08:54:57,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:57,779 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:57,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:57,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:57,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:57,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:57,781 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:57,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:57,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:57,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79069f63[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:57,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:57,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,784 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:57,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:57,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:57,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:57,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:57,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:57,786 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:57,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:57,788 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:57,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:57,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:57,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:57,788 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:57,789 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:57,789 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,790 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:57,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:57,790 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,791 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:57,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:57,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:57,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:57,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:57,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:57,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:57,795 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:57,795 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:57,795 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,795 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:57,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:57,795 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:57,895 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:57,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:57,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:54:57,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:54:57,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:54:57,899 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:54:57,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:57,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:54:57,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:54:57,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:57,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:57,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:57,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:57,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:54:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:57,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:57,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:54:57,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:54:57,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:54:57,902 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:54:57,902 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:54:57,902 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:54:57,902 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:54:57,902 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:54:57,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:57,903 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. started... 2023-04-28 08:54:57,903 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 08:54:57,903 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:57,903 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:57,903 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:57,903 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:57,903 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:57,903 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:57,905 DEBUG [Thread-1587] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:54655 2023-04-28 08:54:57,905 DEBUG [Thread-1587] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:57,905 DEBUG [Thread-1589] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:54655 2023-04-28 08:54:57,905 DEBUG [Thread-1589] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:57,906 DEBUG [Thread-1593] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:54655 2023-04-28 08:54:57,906 DEBUG [Thread-1593] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:57,907 DEBUG [Thread-1585] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:54655 2023-04-28 08:54:57,907 DEBUG [Thread-1585] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:57,907 DEBUG [Thread-1591] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:54655 2023-04-28 08:54:57,907 DEBUG [Thread-1591] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:58,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:58,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. as already flushing 2023-04-28 08:54:58,213 DEBUG [Thread-1576] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:54655 2023-04-28 08:54:58,213 DEBUG [Thread-1576] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:58,213 DEBUG [Thread-1574] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:54655 2023-04-28 08:54:58,213 DEBUG [Thread-1574] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:58,311 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/0040116964264359b6384a30840fa838 2023-04-28 08:54:58,720 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/8f03f7a0163249c8958d9de12c98ce0e 2023-04-28 08:54:59,128 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/0e9d2da945364865abc7f6d8cb581796 2023-04-28 08:54:59,132 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/0040116964264359b6384a30840fa838 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0040116964264359b6384a30840fa838 2023-04-28 08:54:59,136 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0040116964264359b6384a30840fa838, entries=150, sequenceid=490, filesize=11.9 K 2023-04-28 08:54:59,136 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/8f03f7a0163249c8958d9de12c98ce0e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8f03f7a0163249c8958d9de12c98ce0e 2023-04-28 08:54:59,140 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8f03f7a0163249c8958d9de12c98ce0e, entries=150, sequenceid=490, filesize=11.9 K 2023-04-28 08:54:59,140 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/0e9d2da945364865abc7f6d8cb581796 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/0e9d2da945364865abc7f6d8cb581796 2023-04-28 08:54:59,143 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/0e9d2da945364865abc7f6d8cb581796, entries=150, sequenceid=490, filesize=11.9 K 2023-04-28 08:54:59,144 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=13.42 KB/13740 for 4def4a766691cce8b483578b280ed61b in 1241ms, sequenceid=490, compaction requested=true 2023-04-28 08:54:59,144 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:54:59,144 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:59,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:54:59,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:54:59,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:54:59,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:54:59,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:59,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:54:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:59,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:54:59,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b520d2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:54:59,148 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:54:59,148 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,149 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:54:59,149 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:54:59,149 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:54:59,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:54:59,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:54:59,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:59,150 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:54:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:54:59,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:54:59,152 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:54:59,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:54:59,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:54:59,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:54:59,153 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:54:59,154 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:59,154 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:59,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:54:59,154 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:59,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:59,154 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:59,155 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:54:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:54:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:59,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:54:59,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:54:59,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:54:59,158 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:54:59,158 DEBUG [Thread-1584] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,158 DEBUG [Thread-1584] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:54:59,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:54:59,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:54:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:54:59,258 DEBUG [Thread-1584] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:54:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:54:59,538 DEBUG [Thread-1578] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:54655 2023-04-28 08:54:59,538 DEBUG [Thread-1578] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:59,541 DEBUG [Thread-1580] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:54655 2023-04-28 08:54:59,541 DEBUG [Thread-1580] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:59,548 DEBUG [Thread-1582] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:54655 2023-04-28 08:54:59,548 DEBUG [Thread-1582] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 83 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 5555 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 5594 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 5503 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 5585 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 5607 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 08:54:59,548 INFO [Listener at localhost/39717] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 08:54:59,548 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:54655 2023-04-28 08:54:59,548 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:54:59,549 INFO [Listener at localhost/39717] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 08:54:59,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable TestAcidGuarantees 2023-04-28 08:54:59,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 08:54:59,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 08:54:59,552 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672099552"}]},"ts":"1682672099552"} 2023-04-28 08:54:59,553 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 08:54:59,554 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 08:54:59,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4def4a766691cce8b483578b280ed61b, UNASSIGN}] 2023-04-28 08:54:59,556 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4def4a766691cce8b483578b280ed61b, UNASSIGN 2023-04-28 08:54:59,557 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=4def4a766691cce8b483578b280ed61b, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:54:59,557 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672099557"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672099557"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672099557"}]},"ts":"1682672099557"} 2023-04-28 08:54:59,558 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:54:59,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 08:54:59,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 4def4a766691cce8b483578b280ed61b 2023-04-28 08:54:59,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 4def4a766691cce8b483578b280ed61b, disabling compactions & flushes 2023-04-28 08:54:59,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:59,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:59,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. after waiting 0 ms 2023-04-28 08:54:59,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:54:59,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 4def4a766691cce8b483578b280ed61b 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 08:54:59,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=A 2023-04-28 08:54:59,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:59,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=B 2023-04-28 08:54:59,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:59,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4def4a766691cce8b483578b280ed61b, store=C 2023-04-28 08:54:59,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:54:59,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 08:55:00,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7038683f8dd14784bbeda07b98732a43 2023-04-28 08:55:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 08:55:00,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/4dc51280f3364abca6843b06fbb3dda1 2023-04-28 08:55:00,533 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/8eb1a4b3e8a04ea6a5c3249ff9a3b4a3 2023-04-28 08:55:00,537 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/A/7038683f8dd14784bbeda07b98732a43 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7038683f8dd14784bbeda07b98732a43 2023-04-28 08:55:00,540 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7038683f8dd14784bbeda07b98732a43, entries=150, sequenceid=498, filesize=11.9 K 2023-04-28 08:55:00,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/B/4dc51280f3364abca6843b06fbb3dda1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/4dc51280f3364abca6843b06fbb3dda1 2023-04-28 08:55:00,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/4dc51280f3364abca6843b06fbb3dda1, entries=150, sequenceid=498, filesize=11.9 K 2023-04-28 08:55:00,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/.tmp/C/8eb1a4b3e8a04ea6a5c3249ff9a3b4a3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8eb1a4b3e8a04ea6a5c3249ff9a3b4a3 2023-04-28 08:55:00,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8eb1a4b3e8a04ea6a5c3249ff9a3b4a3, entries=150, sequenceid=498, filesize=11.9 K 2023-04-28 08:55:00,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 4def4a766691cce8b483578b280ed61b in 840ms, sequenceid=498, compaction requested=true 2023-04-28 08:55:00,571 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/bb14182abfb643be9dc4715a5ba82e2f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/2bbb77e6fab44789ae23c80a49b610e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/efc85331608d4e72b736e15d5dcd8302, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3ffdf07b2853480db74837102c52f43e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/79e856712c3541a9b6f99982b85d1d4b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7fe861414d5e4b039e4cd7c4cec0f967, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7877705159904126aa8deb64d38351d9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0fd9294501c445bcabd1a9202c0f2e4b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9aedf56880074aee8a54d76b149869b8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/a9e91bb864a94d0596fa6a34752bc41e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4baa38d611844bde8edadfcf18a7e523, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0c36c617fef54732a35189764fb51e62, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/da59727f4cae4e49844b0da1845674cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e51667bd8d044a989651115ea93c3201, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ded396b167ca4df0a317f7244da6f0ff, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ab695027fb1b44908f6d4eea1953494a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9e2b71357879416ba2f55301fe85190f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/cfbc1a1f1da14b328613ce3997f6b1af, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/056cddbc74ae40c187c5ae9bec8cc27a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e88380f79cbc4660ae38dfae8f51f135, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/69c519cd264c4be4905efc581f677f07, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9093f52181d54e1ab51ecfe5fe1a9b5a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/78ec2cdb55a44d34b93797e4b1b81b14, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3b60722361854780beeb6260611377dd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/348c52da61804a74997d633daafb91cc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/6aa9f91ee6474764ac521a7c47a52357, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/910fd916873a46e3bd2639ead0721b1e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/59b8b8215c2e43a590e8fe044c56d959, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4e8bbc105c974c589edcae01da0a9b71, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/8d1c7daf0d074c65953541890bc899eb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/f0f6533cb7424c6faa14ffe89236e159] to archive 2023-04-28 08:55:00,572 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:55:00,573 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/bb14182abfb643be9dc4715a5ba82e2f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/bb14182abfb643be9dc4715a5ba82e2f 2023-04-28 08:55:00,575 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/2bbb77e6fab44789ae23c80a49b610e6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/2bbb77e6fab44789ae23c80a49b610e6 2023-04-28 08:55:00,576 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/efc85331608d4e72b736e15d5dcd8302 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/efc85331608d4e72b736e15d5dcd8302 2023-04-28 08:55:00,577 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3ffdf07b2853480db74837102c52f43e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3ffdf07b2853480db74837102c52f43e 2023-04-28 08:55:00,578 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/79e856712c3541a9b6f99982b85d1d4b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/79e856712c3541a9b6f99982b85d1d4b 2023-04-28 08:55:00,579 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7fe861414d5e4b039e4cd7c4cec0f967 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7fe861414d5e4b039e4cd7c4cec0f967 2023-04-28 08:55:00,580 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7877705159904126aa8deb64d38351d9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7877705159904126aa8deb64d38351d9 2023-04-28 08:55:00,581 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0fd9294501c445bcabd1a9202c0f2e4b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0fd9294501c445bcabd1a9202c0f2e4b 2023-04-28 08:55:00,582 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9aedf56880074aee8a54d76b149869b8 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9aedf56880074aee8a54d76b149869b8 2023-04-28 08:55:00,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/a9e91bb864a94d0596fa6a34752bc41e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/a9e91bb864a94d0596fa6a34752bc41e 2023-04-28 08:55:00,583 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4baa38d611844bde8edadfcf18a7e523 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4baa38d611844bde8edadfcf18a7e523 2023-04-28 08:55:00,584 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0c36c617fef54732a35189764fb51e62 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0c36c617fef54732a35189764fb51e62 2023-04-28 08:55:00,585 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/da59727f4cae4e49844b0da1845674cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/da59727f4cae4e49844b0da1845674cb 2023-04-28 08:55:00,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e51667bd8d044a989651115ea93c3201 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e51667bd8d044a989651115ea93c3201 2023-04-28 08:55:00,586 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ded396b167ca4df0a317f7244da6f0ff to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ded396b167ca4df0a317f7244da6f0ff 2023-04-28 08:55:00,587 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ab695027fb1b44908f6d4eea1953494a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/ab695027fb1b44908f6d4eea1953494a 2023-04-28 08:55:00,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9e2b71357879416ba2f55301fe85190f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9e2b71357879416ba2f55301fe85190f 2023-04-28 08:55:00,588 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/cfbc1a1f1da14b328613ce3997f6b1af to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/cfbc1a1f1da14b328613ce3997f6b1af 2023-04-28 08:55:00,589 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/056cddbc74ae40c187c5ae9bec8cc27a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/056cddbc74ae40c187c5ae9bec8cc27a 2023-04-28 08:55:00,590 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e88380f79cbc4660ae38dfae8f51f135 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/e88380f79cbc4660ae38dfae8f51f135 2023-04-28 08:55:00,591 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/69c519cd264c4be4905efc581f677f07 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/69c519cd264c4be4905efc581f677f07 2023-04-28 08:55:00,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9093f52181d54e1ab51ecfe5fe1a9b5a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/9093f52181d54e1ab51ecfe5fe1a9b5a 2023-04-28 08:55:00,592 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/78ec2cdb55a44d34b93797e4b1b81b14 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/78ec2cdb55a44d34b93797e4b1b81b14 2023-04-28 08:55:00,593 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3b60722361854780beeb6260611377dd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/3b60722361854780beeb6260611377dd 2023-04-28 08:55:00,594 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/348c52da61804a74997d633daafb91cc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/348c52da61804a74997d633daafb91cc 2023-04-28 08:55:00,595 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/6aa9f91ee6474764ac521a7c47a52357 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/6aa9f91ee6474764ac521a7c47a52357 2023-04-28 08:55:00,596 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/910fd916873a46e3bd2639ead0721b1e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/910fd916873a46e3bd2639ead0721b1e 2023-04-28 08:55:00,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/59b8b8215c2e43a590e8fe044c56d959 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/59b8b8215c2e43a590e8fe044c56d959 2023-04-28 08:55:00,597 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4e8bbc105c974c589edcae01da0a9b71 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/4e8bbc105c974c589edcae01da0a9b71 2023-04-28 08:55:00,598 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/8d1c7daf0d074c65953541890bc899eb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/8d1c7daf0d074c65953541890bc899eb 2023-04-28 08:55:00,599 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/f0f6533cb7424c6faa14ffe89236e159 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/f0f6533cb7424c6faa14ffe89236e159 2023-04-28 08:55:00,602 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9c89dad6eba34af181cc54fc1823213f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/99bc39d1e94140288c5275856dfeeac3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/46828a4fb51c4d62b015d7bd876f78bd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b9a21dda1c084a03b92fd1a2abff49a4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7c985c9ad1b34e98969852890b41318e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/44647528fe4a407a8138b59b4493a826, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e92ead9857e046ce96c64242fa788071, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/5d4b8ed22d0543209ead952ab7c8c6c0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8a03e98fbb684f30955d5ff830e9f416, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6816d294edb3463f987980f9d074757e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/14a9e36fc75442028f87307f61ca0ede, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/2fb3e402f6fe47fbb2d98ba608b5562b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/cf77d4c083cb4cfa94d83529c990895f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9d24ced66ff945ec8abec05d9c97ebaa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8559608b5a384445bebb56c016b92ad5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/f05c4964d9514b6a9fcfc6eddd4509d7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/84e7f9d6bfec488daec19a73521c8df1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e7768cc6d43148a4804a50d709ec5f11, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/428b72946e5b4de9b3c309b05b49faeb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8506bcc1e51b4876af38ab865eb4570c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/d498a459c46b41fb954a7eae3cef674c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/a973797e5dce4aceb4920540f63640b7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/3426b9031d4c42f0acb811818ed0266a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/40a0e7ced4fe4aa9a697710db1017dd1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/852f33b6a1d5431c95d8341065102b4e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/81e204af1c7b4a9387f81f330bdfb6de, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6ef3589cb15e49718fc489c91ac2c9c8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7473fa8635c944b69fbfb0be51be5e15, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/0e0c7f04477644feb4e6de13ab725a20, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/c0f74ae88b4a4c2fb8985f1936b9b32e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/38e8301f4883440ead50da13bc2cc46a] to archive 2023-04-28 08:55:00,603 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:55:00,604 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9c89dad6eba34af181cc54fc1823213f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9c89dad6eba34af181cc54fc1823213f 2023-04-28 08:55:00,605 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/99bc39d1e94140288c5275856dfeeac3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/99bc39d1e94140288c5275856dfeeac3 2023-04-28 08:55:00,606 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/46828a4fb51c4d62b015d7bd876f78bd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/46828a4fb51c4d62b015d7bd876f78bd 2023-04-28 08:55:00,607 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b9a21dda1c084a03b92fd1a2abff49a4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b9a21dda1c084a03b92fd1a2abff49a4 2023-04-28 08:55:00,608 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7c985c9ad1b34e98969852890b41318e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7c985c9ad1b34e98969852890b41318e 2023-04-28 08:55:00,609 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/44647528fe4a407a8138b59b4493a826 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/44647528fe4a407a8138b59b4493a826 2023-04-28 08:55:00,610 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e92ead9857e046ce96c64242fa788071 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e92ead9857e046ce96c64242fa788071 2023-04-28 08:55:00,611 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/5d4b8ed22d0543209ead952ab7c8c6c0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/5d4b8ed22d0543209ead952ab7c8c6c0 2023-04-28 08:55:00,612 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8a03e98fbb684f30955d5ff830e9f416 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8a03e98fbb684f30955d5ff830e9f416 2023-04-28 08:55:00,613 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6816d294edb3463f987980f9d074757e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6816d294edb3463f987980f9d074757e 2023-04-28 08:55:00,614 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/14a9e36fc75442028f87307f61ca0ede to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/14a9e36fc75442028f87307f61ca0ede 2023-04-28 08:55:00,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/2fb3e402f6fe47fbb2d98ba608b5562b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/2fb3e402f6fe47fbb2d98ba608b5562b 2023-04-28 08:55:00,615 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/cf77d4c083cb4cfa94d83529c990895f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/cf77d4c083cb4cfa94d83529c990895f 2023-04-28 08:55:00,616 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9d24ced66ff945ec8abec05d9c97ebaa to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/9d24ced66ff945ec8abec05d9c97ebaa 2023-04-28 08:55:00,617 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8559608b5a384445bebb56c016b92ad5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8559608b5a384445bebb56c016b92ad5 2023-04-28 08:55:00,618 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/f05c4964d9514b6a9fcfc6eddd4509d7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/f05c4964d9514b6a9fcfc6eddd4509d7 2023-04-28 08:55:00,619 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/84e7f9d6bfec488daec19a73521c8df1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/84e7f9d6bfec488daec19a73521c8df1 2023-04-28 08:55:00,620 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e7768cc6d43148a4804a50d709ec5f11 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/e7768cc6d43148a4804a50d709ec5f11 2023-04-28 08:55:00,621 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/428b72946e5b4de9b3c309b05b49faeb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/428b72946e5b4de9b3c309b05b49faeb 2023-04-28 08:55:00,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8506bcc1e51b4876af38ab865eb4570c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8506bcc1e51b4876af38ab865eb4570c 2023-04-28 08:55:00,622 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/d498a459c46b41fb954a7eae3cef674c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/d498a459c46b41fb954a7eae3cef674c 2023-04-28 08:55:00,623 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/a973797e5dce4aceb4920540f63640b7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/a973797e5dce4aceb4920540f63640b7 2023-04-28 08:55:00,624 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/3426b9031d4c42f0acb811818ed0266a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/3426b9031d4c42f0acb811818ed0266a 2023-04-28 08:55:00,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/40a0e7ced4fe4aa9a697710db1017dd1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/40a0e7ced4fe4aa9a697710db1017dd1 2023-04-28 08:55:00,625 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/852f33b6a1d5431c95d8341065102b4e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/852f33b6a1d5431c95d8341065102b4e 2023-04-28 08:55:00,626 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/81e204af1c7b4a9387f81f330bdfb6de to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/81e204af1c7b4a9387f81f330bdfb6de 2023-04-28 08:55:00,627 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6ef3589cb15e49718fc489c91ac2c9c8 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/6ef3589cb15e49718fc489c91ac2c9c8 2023-04-28 08:55:00,628 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7473fa8635c944b69fbfb0be51be5e15 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/7473fa8635c944b69fbfb0be51be5e15 2023-04-28 08:55:00,629 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/0e0c7f04477644feb4e6de13ab725a20 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/0e0c7f04477644feb4e6de13ab725a20 2023-04-28 08:55:00,630 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/c0f74ae88b4a4c2fb8985f1936b9b32e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/c0f74ae88b4a4c2fb8985f1936b9b32e 2023-04-28 08:55:00,631 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/38e8301f4883440ead50da13bc2cc46a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/38e8301f4883440ead50da13bc2cc46a 2023-04-28 08:55:00,634 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/666ee73b41ad4748b1590e0b00f3d1fb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e54a83fe66bc4c3db5f9682e68ed2559, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/baac4201786548c09dd77fa2f2e1ec2b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6c4ad65df25c40acbdb6faaa71c99469, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5a92c1b3e1834c7aac2334a990c120c5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7162f7f4dcde48bfab5d8a0bf8ac8713, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/561ae051e1e342a39eee552fd0f8a53a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54012042565548bb8cb83bf0cdf53942, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6343bd49cb8b4487bc126b047f5bb60c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/3cd95ebe1adb4b14b38b78e1721380cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54dec07001684e2ca63523a44dff36c2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e50e3494117b4334b13bc6d16adb6e71, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a21c350faa9645f586b51cbf2d46fd1f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a44794e5caa1441f873a6f32434a4d42, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/25eca1c028694effa820a2d7c831eab5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7d387fa114b84ca3bcbbddb8542ef6b0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/263485668dd5499fa531f13a2da8c631, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/1f5ee6fe11ea4ea6945674d0b42d2980, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/006df79f32c049aaa7be51f2965c7f67, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4e25a6797608407db62f483bed79d1f4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6d6f46d3089147128ad219ce39bd9e29, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8f245813cf4b44cba4e86efbebd21cb6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/67ea3187a3004023a86d40d2bb36503f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4ded6a5c5e774cef8151a5196f3290ca, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/dd5cbf22617847758b84e511348b252f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/2040a2c854b747ab8d2ca243a8aa170a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/016db0d9722d453a8c37d3396e49ea26, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5b86907ea65e486b9d6c9365aebc0afd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e7daac3149d3438a8d3e8488220e5b9f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e0528f29f7344054978e8e30958bbaa2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4429f107ae3f4b15b02c0022fbfde633] to archive 2023-04-28 08:55:00,634 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:55:00,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/666ee73b41ad4748b1590e0b00f3d1fb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/666ee73b41ad4748b1590e0b00f3d1fb 2023-04-28 08:55:00,636 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e54a83fe66bc4c3db5f9682e68ed2559 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e54a83fe66bc4c3db5f9682e68ed2559 2023-04-28 08:55:00,637 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/baac4201786548c09dd77fa2f2e1ec2b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/baac4201786548c09dd77fa2f2e1ec2b 2023-04-28 08:55:00,638 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6c4ad65df25c40acbdb6faaa71c99469 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6c4ad65df25c40acbdb6faaa71c99469 2023-04-28 08:55:00,639 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5a92c1b3e1834c7aac2334a990c120c5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5a92c1b3e1834c7aac2334a990c120c5 2023-04-28 08:55:00,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7162f7f4dcde48bfab5d8a0bf8ac8713 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7162f7f4dcde48bfab5d8a0bf8ac8713 2023-04-28 08:55:00,640 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/561ae051e1e342a39eee552fd0f8a53a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/561ae051e1e342a39eee552fd0f8a53a 2023-04-28 08:55:00,641 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54012042565548bb8cb83bf0cdf53942 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54012042565548bb8cb83bf0cdf53942 2023-04-28 08:55:00,642 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6343bd49cb8b4487bc126b047f5bb60c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6343bd49cb8b4487bc126b047f5bb60c 2023-04-28 08:55:00,643 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/3cd95ebe1adb4b14b38b78e1721380cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/3cd95ebe1adb4b14b38b78e1721380cb 2023-04-28 08:55:00,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54dec07001684e2ca63523a44dff36c2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/54dec07001684e2ca63523a44dff36c2 2023-04-28 08:55:00,644 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e50e3494117b4334b13bc6d16adb6e71 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e50e3494117b4334b13bc6d16adb6e71 2023-04-28 08:55:00,645 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a21c350faa9645f586b51cbf2d46fd1f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a21c350faa9645f586b51cbf2d46fd1f 2023-04-28 08:55:00,646 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a44794e5caa1441f873a6f32434a4d42 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/a44794e5caa1441f873a6f32434a4d42 2023-04-28 08:55:00,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/25eca1c028694effa820a2d7c831eab5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/25eca1c028694effa820a2d7c831eab5 2023-04-28 08:55:00,647 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7d387fa114b84ca3bcbbddb8542ef6b0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/7d387fa114b84ca3bcbbddb8542ef6b0 2023-04-28 08:55:00,648 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/263485668dd5499fa531f13a2da8c631 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/263485668dd5499fa531f13a2da8c631 2023-04-28 08:55:00,649 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/1f5ee6fe11ea4ea6945674d0b42d2980 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/1f5ee6fe11ea4ea6945674d0b42d2980 2023-04-28 08:55:00,650 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/006df79f32c049aaa7be51f2965c7f67 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/006df79f32c049aaa7be51f2965c7f67 2023-04-28 08:55:00,651 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4e25a6797608407db62f483bed79d1f4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4e25a6797608407db62f483bed79d1f4 2023-04-28 08:55:00,652 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6d6f46d3089147128ad219ce39bd9e29 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/6d6f46d3089147128ad219ce39bd9e29 2023-04-28 08:55:00,652 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8f245813cf4b44cba4e86efbebd21cb6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8f245813cf4b44cba4e86efbebd21cb6 2023-04-28 08:55:00,653 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/67ea3187a3004023a86d40d2bb36503f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/67ea3187a3004023a86d40d2bb36503f 2023-04-28 08:55:00,654 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4ded6a5c5e774cef8151a5196f3290ca to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4ded6a5c5e774cef8151a5196f3290ca 2023-04-28 08:55:00,655 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/dd5cbf22617847758b84e511348b252f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/dd5cbf22617847758b84e511348b252f 2023-04-28 08:55:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 08:55:00,656 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/2040a2c854b747ab8d2ca243a8aa170a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/2040a2c854b747ab8d2ca243a8aa170a 2023-04-28 08:55:00,657 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/016db0d9722d453a8c37d3396e49ea26 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/016db0d9722d453a8c37d3396e49ea26 2023-04-28 08:55:00,657 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5b86907ea65e486b9d6c9365aebc0afd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/5b86907ea65e486b9d6c9365aebc0afd 2023-04-28 08:55:00,658 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e7daac3149d3438a8d3e8488220e5b9f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e7daac3149d3438a8d3e8488220e5b9f 2023-04-28 08:55:00,659 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e0528f29f7344054978e8e30958bbaa2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e0528f29f7344054978e8e30958bbaa2 2023-04-28 08:55:00,660 DEBUG [StoreCloser-TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4429f107ae3f4b15b02c0022fbfde633 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/4429f107ae3f4b15b02c0022fbfde633 2023-04-28 08:55:00,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/recovered.edits/501.seqid, newMaxSeqId=501, maxSeqId=1 2023-04-28 08:55:00,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b. 2023-04-28 08:55:00,665 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 4def4a766691cce8b483578b280ed61b: 2023-04-28 08:55:00,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 4def4a766691cce8b483578b280ed61b 2023-04-28 08:55:00,666 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=4def4a766691cce8b483578b280ed61b, regionState=CLOSED 2023-04-28 08:55:00,666 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672100666"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672100666"}]},"ts":"1682672100666"} 2023-04-28 08:55:00,668 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-28 08:55:00,668 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 4def4a766691cce8b483578b280ed61b, server=jenkins-hbase6.apache.org,40909,1682672018517 in 1.1100 sec 2023-04-28 08:55:00,670 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-28 08:55:00,670 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4def4a766691cce8b483578b280ed61b, UNASSIGN in 1.1130 sec 2023-04-28 08:55:00,670 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672100670"}]},"ts":"1682672100670"} 2023-04-28 08:55:00,671 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 08:55:00,672 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 08:55:00,673 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1240 sec 2023-04-28 08:55:01,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-28 08:55:01,656 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-28 08:55:01,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete TestAcidGuarantees 2023-04-28 08:55:01,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:01,659 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:01,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-28 08:55:01,659 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:01,664 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b 2023-04-28 08:55:01,666 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/recovered.edits] 2023-04-28 08:55:01,668 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0040116964264359b6384a30840fa838 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/0040116964264359b6384a30840fa838 2023-04-28 08:55:01,669 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/41ef422e1b334a0689e324268258806c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/41ef422e1b334a0689e324268258806c 2023-04-28 08:55:01,670 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7038683f8dd14784bbeda07b98732a43 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7038683f8dd14784bbeda07b98732a43 2023-04-28 08:55:01,671 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7daa2d4b71e840c2a243fe4470e8c4d6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/A/7daa2d4b71e840c2a243fe4470e8c4d6 2023-04-28 08:55:01,673 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/4dc51280f3364abca6843b06fbb3dda1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/4dc51280f3364abca6843b06fbb3dda1 2023-04-28 08:55:01,674 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8f03f7a0163249c8958d9de12c98ce0e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/8f03f7a0163249c8958d9de12c98ce0e 2023-04-28 08:55:01,675 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b83abd1b9d30415b81319e88406e143e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/b83abd1b9d30415b81319e88406e143e 2023-04-28 08:55:01,676 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/bf9960257abf43feab43fe2c198e206a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/B/bf9960257abf43feab43fe2c198e206a 2023-04-28 08:55:01,678 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/0e9d2da945364865abc7f6d8cb581796 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/0e9d2da945364865abc7f6d8cb581796 2023-04-28 08:55:01,679 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/451ffbaee6ce4b06ae0dbd70c0874e30 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/451ffbaee6ce4b06ae0dbd70c0874e30 2023-04-28 08:55:01,680 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8eb1a4b3e8a04ea6a5c3249ff9a3b4a3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/8eb1a4b3e8a04ea6a5c3249ff9a3b4a3 2023-04-28 08:55:01,681 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e218ec7a92bd46bb881b9571f271aa55 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/C/e218ec7a92bd46bb881b9571f271aa55 2023-04-28 08:55:01,683 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/recovered.edits/501.seqid to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b/recovered.edits/501.seqid 2023-04-28 08:55:01,684 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/4def4a766691cce8b483578b280ed61b 2023-04-28 08:55:01,684 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:55:01,686 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:01,687 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 08:55:01,688 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 08:55:01,689 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:01,689 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 08:55:01,689 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682672101689"}]},"ts":"9223372036854775807"} 2023-04-28 08:55:01,691 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 08:55:01,691 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4def4a766691cce8b483578b280ed61b, NAME => 'TestAcidGuarantees,,1682672075740.4def4a766691cce8b483578b280ed61b.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 08:55:01,691 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 08:55:01,691 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682672101691"}]},"ts":"9223372036854775807"} 2023-04-28 08:55:01,692 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 08:55:01,694 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:01,694 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 37 msec 2023-04-28 08:55:01,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-28 08:55:01,761 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-28 08:55:01,772 INFO [Listener at localhost/39717] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 236), OpenFileDescriptor=556 (was 555) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=423 (was 436), ProcessCount=170 (was 172), AvailableMemoryMB=6080 (was 6109) 2023-04-28 08:55:01,784 INFO [Listener at localhost/39717] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=556, MaxFileDescriptor=60000, SystemLoadAverage=423, ProcessCount=170, AvailableMemoryMB=6080 2023-04-28 08:55:01,786 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:55:01,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:55:01,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:01,788 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 08:55:01,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-28 08:55:01,789 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 08:55:01,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 08:55:01,790 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:01,790 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 empty. 2023-04-28 08:55:01,791 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:01,791 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:55:01,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 08:55:02,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 08:55:02,198 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:55:02,199 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5fb523e858d6851c02bb0f82113765f7, NAME => 'TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp 2023-04-28 08:55:02,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 08:55:02,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:55:02,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5fb523e858d6851c02bb0f82113765f7, disabling compactions & flushes 2023-04-28 08:55:02,605 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. after waiting 0 ms 2023-04-28 08:55:02,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,605 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,605 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:02,607 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 08:55:02,608 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672102608"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672102608"}]},"ts":"1682672102608"} 2023-04-28 08:55:02,609 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 08:55:02,609 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 08:55:02,609 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672102609"}]},"ts":"1682672102609"} 2023-04-28 08:55:02,610 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 08:55:02,613 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, ASSIGN}] 2023-04-28 08:55:02,615 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, ASSIGN 2023-04-28 08:55:02,615 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=false 2023-04-28 08:55:02,766 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:02,767 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672102766"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672102766"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672102766"}]},"ts":"1682672102766"} 2023-04-28 08:55:02,768 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:55:02,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 08:55:02,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5fb523e858d6851c02bb0f82113765f7, NAME => 'TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:55:02,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:55:02,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,924 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,925 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A 2023-04-28 08:55:02,925 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A 2023-04-28 08:55:02,926 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:02,926 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5fb523e858d6851c02bb0f82113765f7 columnFamilyName A 2023-04-28 08:55:02,926 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(310): Store=5fb523e858d6851c02bb0f82113765f7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:02,926 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,927 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B 2023-04-28 08:55:02,927 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B 2023-04-28 08:55:02,927 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:02,928 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5fb523e858d6851c02bb0f82113765f7 columnFamilyName B 2023-04-28 08:55:02,928 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(310): Store=5fb523e858d6851c02bb0f82113765f7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:02,928 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,929 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C 2023-04-28 08:55:02,929 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C 2023-04-28 08:55:02,929 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:02,929 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5fb523e858d6851c02bb0f82113765f7 columnFamilyName C 2023-04-28 08:55:02,930 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(310): Store=5fb523e858d6851c02bb0f82113765f7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:02,930 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:55:02,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:02,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:55:02,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 5fb523e858d6851c02bb0f82113765f7; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58814609, jitterRate=-0.12359403073787689}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:55:02,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:02,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., pid=37, masterSystemTime=1682672102919 2023-04-28 08:55:02,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:02,937 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:02,937 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672102937"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672102937"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672102937"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672102937"}]},"ts":"1682672102937"} 2023-04-28 08:55:02,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-28 08:55:02,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 in 170 msec 2023-04-28 08:55:02,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-28 08:55:02,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, ASSIGN in 326 msec 2023-04-28 08:55:02,941 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 08:55:02,941 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672102941"}]},"ts":"1682672102941"} 2023-04-28 08:55:02,942 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 08:55:02,944 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 08:55:02,944 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-04-28 08:55:03,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-28 08:55:03,894 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-28 08:55:03,894 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x49b5bd0d to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:03,899 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bcfc1e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:03,901 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:03,903 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:03,904 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 08:55:03,905 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:35832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 08:55:03,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:55:03,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$12(2454): Client=jenkins//172.31.2.20 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:55:03,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:04,317 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 08:55:04,317 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:55:04,318 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 08:55:04,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 08:55:04,322 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, REOPEN/MOVE}] 2023-04-28 08:55:04,322 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, REOPEN/MOVE 2023-04-28 08:55:04,323 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,323 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672104323"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672104323"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672104323"}]},"ts":"1682672104323"} 2023-04-28 08:55:04,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:55:04,476 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 5fb523e858d6851c02bb0f82113765f7, disabling compactions & flushes 2023-04-28 08:55:04,477 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. after waiting 0 ms 2023-04-28 08:55:04,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 08:55:04,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,482 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:04,482 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5fb523e858d6851c02bb0f82113765f7 to self. 2023-04-28 08:55:04,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,484 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=CLOSED 2023-04-28 08:55:04,484 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672104484"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672104484"}]},"ts":"1682672104484"} 2023-04-28 08:55:04,486 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-28 08:55:04,486 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 in 161 msec 2023-04-28 08:55:04,487 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=true 2023-04-28 08:55:04,637 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,637 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672104637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672104637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672104637"}]},"ts":"1682672104637"} 2023-04-28 08:55:04,639 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:55:04,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5fb523e858d6851c02bb0f82113765f7, NAME => 'TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:55:04,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:55:04,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,795 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,796 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A 2023-04-28 08:55:04,796 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A 2023-04-28 08:55:04,797 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:04,797 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5fb523e858d6851c02bb0f82113765f7 columnFamilyName A 2023-04-28 08:55:04,798 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(310): Store=5fb523e858d6851c02bb0f82113765f7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:04,798 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,799 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B 2023-04-28 08:55:04,799 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B 2023-04-28 08:55:04,799 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:04,799 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5fb523e858d6851c02bb0f82113765f7 columnFamilyName B 2023-04-28 08:55:04,800 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(310): Store=5fb523e858d6851c02bb0f82113765f7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:04,800 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,801 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C 2023-04-28 08:55:04,801 DEBUG [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C 2023-04-28 08:55:04,801 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:04,801 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5fb523e858d6851c02bb0f82113765f7 columnFamilyName C 2023-04-28 08:55:04,801 INFO [StoreOpener-5fb523e858d6851c02bb0f82113765f7-1] regionserver.HStore(310): Store=5fb523e858d6851c02bb0f82113765f7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:04,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:55:04,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 5fb523e858d6851c02bb0f82113765f7; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62881784, jitterRate=-0.06298840045928955}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:55:04,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:04,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., pid=42, masterSystemTime=1682672104790 2023-04-28 08:55:04,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,808 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,808 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672104808"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672104808"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672104808"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672104808"}]},"ts":"1682672104808"} 2023-04-28 08:55:04,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-28 08:55:04,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 in 170 msec 2023-04-28 08:55:04,811 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-28 08:55:04,811 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, REOPEN/MOVE in 488 msec 2023-04-28 08:55:04,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-28 08:55:04,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 493 msec 2023-04-28 08:55:04,815 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-28 08:55:04,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-28 08:55:04,817 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5c8d927d to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,824 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38341134, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,825 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x419b69bf to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,833 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71b5058b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,834 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x2c6cac96 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,838 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3620ee35, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,839 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x06a0df34 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,845 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe1db6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,845 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x10ee5f06 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,850 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10377b50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,851 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x4366fcdd to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,858 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68a0e21a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,858 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x6a35a28d to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,865 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ec12b02, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,865 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5a1d4246 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,876 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d7ba242, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,877 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x0eb8f142 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,885 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e079da1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,885 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5ce79633 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:04,893 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@159c5e0c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:04,909 DEBUG [hconnection-0x25e5c999-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,910 DEBUG [hconnection-0x779e0256-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,911 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48378, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:04,912 DEBUG [hconnection-0x53e2c6a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,913 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48382, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,913 DEBUG [hconnection-0x742847ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,914 DEBUG [hconnection-0x716decbe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,915 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,915 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48408, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,919 DEBUG [hconnection-0x7efbe3b3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,919 DEBUG [hconnection-0xabed19e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,924 DEBUG [hconnection-0x17fc61eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,924 DEBUG [hconnection-0x1ba8fb2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,927 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,928 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48432, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,932 DEBUG [hconnection-0x5f0bf7e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:04,933 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,933 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,935 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48450, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,935 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:48456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:04,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:04,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:04,939 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:04,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:04,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:04,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:04,940 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,940 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:04,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,942 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:04,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:04,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:04,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:04,945 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,945 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:04,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:04,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:04,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:04,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:04,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:55:04,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:04,958 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:04,958 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:04,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:04,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:04,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:04,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:04,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:04,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:04,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:04,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:04,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:04,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:04,962 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:04,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:04,962 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:04,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a8bc11[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:04,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:04,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,966 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:04,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:04,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:04,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:04,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:04,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:04,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:04,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:04,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:04,970 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:04,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:04,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:04,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:04,971 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:04,972 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,972 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,972 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:04,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:04,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:04,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:04,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:04,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:04,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672164971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,973 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:04,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:04,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672164971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:04,974 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:04,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:04,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:04,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:04,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:04,980 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:04,980 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:04,980 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:04,980 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:04,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:04,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:04,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672164972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:04,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:04,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672164973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:04,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:04,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672164973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,000 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428626a6391a01e4e79b18730dd5dced8ac to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428626a6391a01e4e79b18730dd5dced8ac 2023-04-28 08:55:05,013 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/940844700ec1435d8913551657b6861e 2023-04-28 08:55:05,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672165074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672165074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,080 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672165082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672165082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672165082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,091 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,095 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,095 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:05,099 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:05,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:05,099 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,100 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:05,100 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:05,102 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,102 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,102 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e8d186[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:05,103 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:05,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,105 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,105 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,105 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:05,105 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:05,105 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:05,105 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:05,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:05,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:05,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:05,111 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:05,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:05,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:05,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:05,111 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:05,113 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,113 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,113 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,113 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,115 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,117 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:05,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,117 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,117 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:05,118 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:05,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:05,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,218 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,221 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,225 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:05,226 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:05,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:05,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:05,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:05,227 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,227 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,228 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,228 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:05,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e58af1f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:05,229 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:05,229 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,230 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,230 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:05,230 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:05,230 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:05,230 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,230 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:05,231 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,232 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:05,232 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,232 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:05,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:05,234 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:05,234 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:05,234 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:05,234 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:05,234 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:05,238 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,238 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,238 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,239 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,240 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,245 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,245 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,245 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:05,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:05,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,245 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,246 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:05,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,246 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:05,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,246 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672165277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672165278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672165285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672165287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672165287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,346 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,349 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,351 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,351 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,353 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,353 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,353 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:05,353 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:05,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:05,356 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:05,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1095a1b6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:05,357 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:05,357 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:05,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:05,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:05,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:05,360 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,360 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:05,360 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,360 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:05,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:05,362 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:05,362 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:05,362 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:05,362 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:05,362 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:05,364 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,364 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,364 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,364 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:05,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:05,370 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:05,371 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:05,371 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:05,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/025bf7226e1843a18b0feecf885b86ba 2023-04-28 08:55:05,471 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,475 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,479 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,479 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,479 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:05,479 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:05,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:05,481 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,481 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,481 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,481 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:05,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:05,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cd1df81[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:05,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:05,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,484 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:05,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:05,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:05,484 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,484 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:05,486 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:05,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:05,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:05,489 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:05,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:05,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:05,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:05,489 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:05,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,492 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,495 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:05,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:05,495 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:05,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672165579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672165584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672165588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672165588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:05,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672165590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,595 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,599 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,600 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,601 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,601 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,601 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,603 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:05,603 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,603 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:05,603 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:05,603 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,603 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:05,603 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:05,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,605 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,606 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:05,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d0597c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:05,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:05,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,608 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,608 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:05,608 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:05,609 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:05,609 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:05,610 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:05,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,610 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:05,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:05,612 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:05,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:05,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:05,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:05,613 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:05,614 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,614 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,614 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,615 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:05,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:05,620 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:05,620 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:05,720 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,723 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,724 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,724 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,727 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,727 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,727 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,728 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,729 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,729 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,729 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:05,729 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:05,729 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:05,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,731 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,731 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:05,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525b210e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:05,732 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:05,732 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,734 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,734 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:05,734 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:05,734 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:05,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,735 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:05,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,736 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:05,736 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,736 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:05,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:05,738 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:05,738 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:05,738 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:05,738 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:05,738 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:05,740 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,740 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,740 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,740 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:05,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:05,744 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,745 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:05,745 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,745 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,745 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,745 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:05,745 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:05,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/fbd3c6c5cba3460a810a1dd22013d45f 2023-04-28 08:55:05,845 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,851 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/940844700ec1435d8913551657b6861e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/940844700ec1435d8913551657b6861e 2023-04-28 08:55:05,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,851 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,853 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,853 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,854 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,854 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/940844700ec1435d8913551657b6861e, entries=150, sequenceid=18, filesize=25.1 K 2023-04-28 08:55:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/025bf7226e1843a18b0feecf885b86ba as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/025bf7226e1843a18b0feecf885b86ba 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,856 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,856 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,856 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:05,856 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,856 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:05,857 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:05,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,858 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:05,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:05,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f7eb2e2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:05,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:05,860 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,861 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/025bf7226e1843a18b0feecf885b86ba, entries=150, sequenceid=18, filesize=11.6 K 2023-04-28 08:55:05,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:05,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:05,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:05,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:05,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/fbd3c6c5cba3460a810a1dd22013d45f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/fbd3c6c5cba3460a810a1dd22013d45f 2023-04-28 08:55:05,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:05,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,863 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,863 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:05,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:05,865 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:05,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:05,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:05,865 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:05,865 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:05,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/fbd3c6c5cba3460a810a1dd22013d45f, entries=150, sequenceid=18, filesize=11.6 K 2023-04-28 08:55:05,868 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,868 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=114.05 KB/116790 for 5fb523e858d6851c02bb0f82113765f7 in 916ms, sequenceid=18, compaction requested=false 2023-04-28 08:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:05,868 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:05,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,870 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,872 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,872 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:05,872 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,872 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:05,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:05,872 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:05,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:05,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:05,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:05,873 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,873 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:05,873 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:05,873 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,873 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:05,873 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,973 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:05,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:05,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:05,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:05,977 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:05,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:05,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:05,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,977 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,978 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:05,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:05,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:05,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:05,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:05,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:05,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:05,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:05,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:05,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:05,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:05,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:05,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:05,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:05,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:05,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:05,980 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:05,980 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:05,997 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042851d0615e407747d6be9e04551f961600 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042851d0615e407747d6be9e04551f961600 2023-04-28 08:55:06,015 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/44bd1633f80646f58da0a344bf47ea81 2023-04-28 08:55:06,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:06,087 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:06,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672166097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672166097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672166103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672166104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672166104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,199 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 08:55:06,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672166207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672166211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672166211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672166212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672166411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672166413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672166414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672166419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,426 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/fa3176ddbf244df095aa5ce59e28e4f7 2023-04-28 08:55:06,448 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/8fa64e7eec5b4648aecfa3d843fb0053 2023-04-28 08:55:06,455 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/44bd1633f80646f58da0a344bf47ea81 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/44bd1633f80646f58da0a344bf47ea81 2023-04-28 08:55:06,459 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/44bd1633f80646f58da0a344bf47ea81, entries=150, sequenceid=38, filesize=25.1 K 2023-04-28 08:55:06,460 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/fa3176ddbf244df095aa5ce59e28e4f7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fa3176ddbf244df095aa5ce59e28e4f7 2023-04-28 08:55:06,465 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fa3176ddbf244df095aa5ce59e28e4f7, entries=150, sequenceid=38, filesize=11.6 K 2023-04-28 08:55:06,466 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/8fa64e7eec5b4648aecfa3d843fb0053 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/8fa64e7eec5b4648aecfa3d843fb0053 2023-04-28 08:55:06,475 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/8fa64e7eec5b4648aecfa3d843fb0053, entries=150, sequenceid=38, filesize=11.6 K 2023-04-28 08:55:06,476 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5fb523e858d6851c02bb0f82113765f7 in 496ms, sequenceid=38, compaction requested=false 2023-04-28 08:55:06,476 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:06,476 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:06,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:06,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:06,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:06,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:06,478 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:06,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:06,478 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:06,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,479 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:06,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:06,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:06,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:06,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc1455b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:06,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:06,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,483 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:06,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:06,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:06,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:06,484 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:06,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:06,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:06,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:06,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:06,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:06,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:06,486 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:06,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:06,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:06,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:06,487 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:06,488 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,488 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:06,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:06,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,488 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:06,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:06,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:06,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:06,490 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:06,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:06,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:06,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:06,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:06,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:06,494 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:06,494 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:06,494 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:06,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:06,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:06,494 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,594 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:06,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:06,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:06,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:06,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:06,597 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:06,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:06,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:06,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:06,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:06,600 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:06,600 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:06,600 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,600 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:06,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:06,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:06,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:06,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:06,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:06,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:06,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:06,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:06,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:06,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:06,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:06,602 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:06,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:06,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:06,602 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:55:06,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:06,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:06,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:06,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:06,602 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:06,603 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:06,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:06,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:06,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672166742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672166743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672106729, max timestamp=1682672106749Num uniques 859; 2023-04-28 08:55:06,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672106729, max timestamp=1682672106749Num uniques 859; 2023-04-28 08:55:06,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:06,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:06,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672106729, max timestamp=1682672106749Num uniques 859; 2023-04-28 08:55:06,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:06,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672166802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672166803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672166851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672166851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672166910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:06,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:06,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672166916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,021 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042872bdafd9ce254b8c833e44f1673d9e10 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042872bdafd9ce254b8c833e44f1673d9e10 2023-04-28 08:55:07,030 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/836e322f0ba8456cb6cb65b65999f889 2023-04-28 08:55:07,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672167057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672167057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672167109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672167114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672167123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672167363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672167363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,424 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672167419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672167429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,447 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/e65cdb8513fd492faf498fc2c2df1c8a 2023-04-28 08:55:07,857 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/7a81843602924aef97fac0dcaed71953 2023-04-28 08:55:07,861 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/836e322f0ba8456cb6cb65b65999f889 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/836e322f0ba8456cb6cb65b65999f889 2023-04-28 08:55:07,864 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/836e322f0ba8456cb6cb65b65999f889, entries=150, sequenceid=50, filesize=25.1 K 2023-04-28 08:55:07,865 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/e65cdb8513fd492faf498fc2c2df1c8a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e65cdb8513fd492faf498fc2c2df1c8a 2023-04-28 08:55:07,869 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e65cdb8513fd492faf498fc2c2df1c8a, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 08:55:07,870 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/7a81843602924aef97fac0dcaed71953 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7a81843602924aef97fac0dcaed71953 2023-04-28 08:55:07,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672167869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672167870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,873 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7a81843602924aef97fac0dcaed71953, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 08:55:07,874 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 5fb523e858d6851c02bb0f82113765f7 in 1272ms, sequenceid=50, compaction requested=true 2023-04-28 08:55:07,874 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:07,874 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:07,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:07,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:07,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:07,874 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:07,878 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:07,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:07,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:07,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:07,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:07,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:07,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621f826[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:07,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:07,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,882 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:07,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:07,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:07,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:07,892 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:07,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:07,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:07,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:07,895 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:07,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:07,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:07,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:07,895 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:07,896 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:07,896 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:07,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:07,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:07,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:07,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:07,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:07,898 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:07,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:07,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:07,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:07,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:07,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:07,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:07,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:07,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:07,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:07,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:07,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:07,901 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:07,901 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:07,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:07,932 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-28 08:55:07,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:07,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:07,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:07,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:07,933 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:07,933 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:07,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672167960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:07,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:07,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672167964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,001 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,004 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,006 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,008 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,011 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d79fabd[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,014 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,014 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,014 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,015 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,015 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,015 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,019 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,019 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,019 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,019 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,019 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,023 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,026 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:08,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,027 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,027 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672168065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672168076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,127 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,130 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,130 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,130 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,130 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,131 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,132 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,132 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,132 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,132 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,133 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,133 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,134 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,134 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,135 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,135 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,136 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,136 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170ab052[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,138 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,139 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,138 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,139 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,139 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,141 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,141 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,143 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,143 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,143 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,143 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,143 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,145 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,145 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,145 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,145 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:08,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,148 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,149 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,149 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,249 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,252 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,253 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,254 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,257 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,257 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6016d475[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,269 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,269 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,269 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,271 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,271 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672168268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,273 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,275 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:08,277 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,277 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672168284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,345 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042804cf01986495408ebc9e278a6dcce483 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042804cf01986495408ebc9e278a6dcce483 2023-04-28 08:55:08,377 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,380 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,385 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,385 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,385 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,385 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,387 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69fff099[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,390 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,390 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,390 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,392 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,394 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,394 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,397 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,397 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,401 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,401 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:08,401 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,402 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,402 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,402 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,402 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,402 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,402 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,402 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,402 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,402 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,502 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,506 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,508 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,508 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,509 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,509 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b914309[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,514 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,516 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,516 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,516 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,516 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,517 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,519 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,520 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,520 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,520 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,520 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,521 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,521 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,521 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,525 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,525 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,525 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672168575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672168588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,625 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,629 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,632 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,632 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,633 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,633 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,634 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,634 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,634 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,634 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,637 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,637 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,637 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69dbfe09[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,638 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,642 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,642 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,642 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,645 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,646 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,649 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,661 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,661 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,751 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/590a6b534d514eeeb0f1e685f2456d70 2023-04-28 08:55:08,761 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,762 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/ded4cedac0274a22974108aca2ebc0b5 2023-04-28 08:55:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,767 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,769 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,769 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,771 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,771 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,772 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,772 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,772 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,775 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,775 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,775 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,775 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,775 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fb83a3b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,787 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,789 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,792 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,793 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,793 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,793 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,793 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,799 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,800 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,800 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:08,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672168879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:08,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672168882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,900 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:08,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:08,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:08,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:08,904 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:08,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:08,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:08,905 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,905 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,908 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,908 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:08,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:08,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:08,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:08,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:08,910 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:08,910 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:08,910 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:08,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:08,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:08,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:08,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:08,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:08,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:08,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:08,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:08,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:08,917 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,917 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:08,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:08,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29556e6d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:08,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:08,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:08,926 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:08,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:08,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:08,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,928 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:08,928 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,928 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:08,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:08,930 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:08,930 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:08,930 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:08,930 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:08,930 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:08,933 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,933 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:08,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,934 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:08,934 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:08,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:08,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:08,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:08,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:08,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:08,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:08,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:08,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:08,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:08,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:08,939 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,939 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:08,940 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:08,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:08,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:08,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:08,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:09,040 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:09,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:09,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:09,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:09,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:09,043 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:09,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:09,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:09,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:09,044 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,044 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,045 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,045 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:09,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:09,045 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,045 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:09,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:09,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:09,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:09,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:09,046 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:09,046 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:09,046 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:09,046 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:09,046 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:09,046 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,047 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:09,047 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:09,047 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:09,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:09,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:09,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:09,048 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,048 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:09,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:09,049 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,049 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:09,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:09,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8fde945[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:09,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:09,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,052 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,052 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:09,052 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,052 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:09,052 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:09,052 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:09,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:09,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:09,053 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:09,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:09,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:09,056 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:09,056 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:09,056 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:09,056 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:09,056 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:09,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:09,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:09,058 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:09,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:09,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:09,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:09,064 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,064 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:09,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,091 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:09,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672169086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:09,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672169098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:09,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672169119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,126 DEBUG [Thread-2275] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:09,164 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:09,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:09,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:09,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:09,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:09,168 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:09,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:09,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:09,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:09,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,170 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:09,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:09,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:09,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:09,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:09,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:09,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:09,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:09,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:09,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:09,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:09,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:09,172 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,172 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:09,172 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:09,172 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:09,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:09,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:09,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:09,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:09,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:09,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:09,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6075e5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:09,175 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:09,175 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,177 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:09,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:09,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:09,177 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,177 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:09,178 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:09,178 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:09,178 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,178 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:09,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:09,181 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:09,181 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:09,181 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:09,181 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:09,181 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:09,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/03cc31afcda64c0397b7cf6a5a9b8c12 2023-04-28 08:55:09,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:09,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:09,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:09,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:09,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,186 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/590a6b534d514eeeb0f1e685f2456d70 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/590a6b534d514eeeb0f1e685f2456d70 2023-04-28 08:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:09,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:09,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:09,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:09,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:09,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,189 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:09,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:09,190 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:09,190 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:09,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:09,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:09,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:09,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/590a6b534d514eeeb0f1e685f2456d70, entries=200, sequenceid=86, filesize=31.9 K 2023-04-28 08:55:09,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/ded4cedac0274a22974108aca2ebc0b5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ded4cedac0274a22974108aca2ebc0b5 2023-04-28 08:55:09,198 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ded4cedac0274a22974108aca2ebc0b5, entries=150, sequenceid=86, filesize=11.6 K 2023-04-28 08:55:09,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/03cc31afcda64c0397b7cf6a5a9b8c12 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/03cc31afcda64c0397b7cf6a5a9b8c12 2023-04-28 08:55:09,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/03cc31afcda64c0397b7cf6a5a9b8c12, entries=150, sequenceid=86, filesize=11.6 K 2023-04-28 08:55:09,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 5fb523e858d6851c02bb0f82113765f7 in 1275ms, sequenceid=86, compaction requested=true 2023-04-28 08:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:09,207 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:09,207 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:09,207 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:09,208 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:09,208 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:09,208 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:09,209 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:09,209 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:09,209 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,209 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,209 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/940844700ec1435d8913551657b6861e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/44bd1633f80646f58da0a344bf47ea81, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/836e322f0ba8456cb6cb65b65999f889, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/590a6b534d514eeeb0f1e685f2456d70] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=107.3 K 2023-04-28 08:55:09,209 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/025bf7226e1843a18b0feecf885b86ba, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fa3176ddbf244df095aa5ce59e28e4f7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e65cdb8513fd492faf498fc2c2df1c8a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ded4cedac0274a22974108aca2ebc0b5] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=46.4 K 2023-04-28 08:55:09,209 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 940844700ec1435d8913551657b6861e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682672104942 2023-04-28 08:55:09,209 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 025bf7226e1843a18b0feecf885b86ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682672104942 2023-04-28 08:55:09,209 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 44bd1633f80646f58da0a344bf47ea81, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682672104970 2023-04-28 08:55:09,209 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting fa3176ddbf244df095aa5ce59e28e4f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682672104970 2023-04-28 08:55:09,210 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e65cdb8513fd492faf498fc2c2df1c8a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672106102 2023-04-28 08:55:09,210 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 836e322f0ba8456cb6cb65b65999f889, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672106102 2023-04-28 08:55:09,211 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ded4cedac0274a22974108aca2ebc0b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682672106801 2023-04-28 08:55:09,212 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 590a6b534d514eeeb0f1e685f2456d70, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682672106781 2023-04-28 08:55:09,221 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#314 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:09,224 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#315 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:09,290 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:09,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:09,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:09,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:09,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:09,293 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:09,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:09,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:09,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:09,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,296 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,296 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:09,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:09,296 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:09,296 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:09,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:09,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:09,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:09,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:09,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:09,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:09,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:09,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:09,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:09,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:09,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:09,298 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:55:09,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:09,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:09,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:09,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:09,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:09,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:09,318 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042874055bcb515a41c2ac55b7a4a93a6aa7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042874055bcb515a41c2ac55b7a4a93a6aa7 2023-04-28 08:55:09,324 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/ae0050181c6a489fb636b3dd33a8602f 2023-04-28 08:55:09,641 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/c49dfbcde51a44d780947cf585aa6155 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49dfbcde51a44d780947cf585aa6155 2023-04-28 08:55:09,647 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into c49dfbcde51a44d780947cf585aa6155(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:09,647 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:09,647 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=12, startTime=1682672109207; duration=0sec 2023-04-28 08:55:09,647 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:09,648 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:09,649 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:09,650 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:09,650 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:09,650 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/fbd3c6c5cba3460a810a1dd22013d45f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/8fa64e7eec5b4648aecfa3d843fb0053, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7a81843602924aef97fac0dcaed71953, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/03cc31afcda64c0397b7cf6a5a9b8c12] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=46.4 K 2023-04-28 08:55:09,650 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting fbd3c6c5cba3460a810a1dd22013d45f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1682672104942 2023-04-28 08:55:09,651 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8fa64e7eec5b4648aecfa3d843fb0053, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1682672104970 2023-04-28 08:55:09,659 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7a81843602924aef97fac0dcaed71953, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672106102 2023-04-28 08:55:09,659 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 03cc31afcda64c0397b7cf6a5a9b8c12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682672106801 2023-04-28 08:55:09,668 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#318 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:09,735 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/c49ec9d1b65e4efe9bb661604829165c 2023-04-28 08:55:10,055 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/3520118f451949199b8a8f0ceb88f4aa as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3520118f451949199b8a8f0ceb88f4aa 2023-04-28 08:55:10,059 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into 3520118f451949199b8a8f0ceb88f4aa(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:10,060 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:10,060 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=12, startTime=1682672109207; duration=0sec 2023-04-28 08:55:10,060 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:10,077 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/aa2db71d3fc84126acc158fbca44b861 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/aa2db71d3fc84126acc158fbca44b861 2023-04-28 08:55:10,081 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into aa2db71d3fc84126acc158fbca44b861(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:10,081 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:10,081 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=12, startTime=1682672109207; duration=0sec 2023-04-28 08:55:10,081 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:10,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:10,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:10,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672110141, max timestamp=1682672110150Num uniques 859; 2023-04-28 08:55:10,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:10,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672110141, max timestamp=1682672110150Num uniques 859; 2023-04-28 08:55:10,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672110141, max timestamp=1682672110150Num uniques 859; 2023-04-28 08:55:10,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:10,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:10,155 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/3c2ed61ef2ef448e800e744164f29589 2023-04-28 08:55:10,159 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/ae0050181c6a489fb636b3dd33a8602f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ae0050181c6a489fb636b3dd33a8602f 2023-04-28 08:55:10,164 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ae0050181c6a489fb636b3dd33a8602f, entries=150, sequenceid=97, filesize=25.1 K 2023-04-28 08:55:10,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/c49ec9d1b65e4efe9bb661604829165c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49ec9d1b65e4efe9bb661604829165c 2023-04-28 08:55:10,168 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49ec9d1b65e4efe9bb661604829165c, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 08:55:10,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/3c2ed61ef2ef448e800e744164f29589 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3c2ed61ef2ef448e800e744164f29589 2023-04-28 08:55:10,178 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3c2ed61ef2ef448e800e744164f29589, entries=150, sequenceid=97, filesize=11.6 K 2023-04-28 08:55:10,178 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=80.51 KB/82440 for 5fb523e858d6851c02bb0f82113765f7 in 880ms, sequenceid=97, compaction requested=false 2023-04-28 08:55:10,179 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:10,179 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:10,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:10,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:10,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:10,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:10,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=87.22 KB heapSize=240.84 KB 2023-04-28 08:55:10,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:10,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:10,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:10,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:10,179 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:10,179 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:10,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,180 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:10,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4077e9f0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:10,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:10,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,186 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:10,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:10,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:10,187 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:10,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,188 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:10,188 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,188 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:10,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:10,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:10,192 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:10,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:10,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:10,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:10,192 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:10,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,196 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:10,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:10,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,202 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:10,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,202 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:10,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672170224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672170229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,302 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:10,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:10,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:10,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:10,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:10,305 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:10,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:10,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:10,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,307 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,307 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,308 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,308 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:10,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:10,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:10,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:10,309 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,309 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:10,309 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:10,309 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:10,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:10,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,311 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,311 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,311 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:10,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:10,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d3d195[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:10,313 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:10,313 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:10,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:10,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:10,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:10,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:10,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:10,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:10,324 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:10,325 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:10,325 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:10,325 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:10,325 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:10,327 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,327 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,328 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,329 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:10,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:10,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,332 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:10,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,332 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:10,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:10,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672170330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672170337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,432 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:10,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:10,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:10,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:10,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:10,435 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:10,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:10,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:10,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:10,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:10,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:10,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:10,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:10,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:10,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:10,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:10,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:10,442 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,442 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:10,442 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:10,442 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:10,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:10,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:10,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:10,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:10,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:10,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2583af4b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:10,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:10,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:10,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:10,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:10,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:10,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:10,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:10,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:10,450 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:10,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:10,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:10,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:10,451 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:10,452 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,452 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,452 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,454 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:10,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,456 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:10,456 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:10,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672170534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672170542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,556 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:10,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:10,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:10,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:10,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:10,559 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:10,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:10,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:10,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,561 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:10,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:10,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:10,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:10,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:10,563 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,563 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:10,563 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:10,563 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:10,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:10,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:10,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18f12943[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:10,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:10,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:10,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:10,569 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:10,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:10,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:10,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:10,572 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:10,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:10,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:10,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:10,572 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:10,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:10,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,579 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:10,579 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,608 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d97008e8a32f4337954a7b44e4a6a2af to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d97008e8a32f4337954a7b44e4a6a2af 2023-04-28 08:55:10,679 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:10,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:10,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:10,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:10,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:10,682 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:10,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:10,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:10,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,684 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,685 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:10,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:10,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:10,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:10,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:10,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:10,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:10,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:10,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:10,687 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,687 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:10,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67b67bc4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:10,689 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:10,689 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:10,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:10,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:10,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:10,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:10,693 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:10,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:10,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:10,695 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:10,695 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:10,695 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:10,695 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:10,695 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:10,697 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,697 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,697 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,697 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,698 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:10,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:10,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:10,702 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,703 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:10,703 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:10,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,803 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:10,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:10,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:10,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:10,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:10,806 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:10,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:10,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:10,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,808 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,808 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,808 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:10,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:10,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:10,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:10,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:10,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:10,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:10,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:10,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:10,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:10,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:10,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:10,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:10,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:10,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b3551de[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:10,818 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:10,818 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,819 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,819 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:10,819 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:10,819 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:10,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:10,820 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:10,820 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,820 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:10,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:10,823 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:10,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:10,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:10,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:10,823 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:10,825 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,825 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,825 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,826 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,827 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:10,835 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:10,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:10,835 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:10,835 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:10,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672170840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672170849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672170888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,896 DEBUG [Thread-2277] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:10,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:10,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672170898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,905 DEBUG [Thread-2273] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:10,935 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:10,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:10,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:10,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:10,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:10,942 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:10,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:10,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:10,943 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,943 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,944 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:10,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:10,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:10,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:10,947 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,947 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:10,947 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:10,947 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:10,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:10,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:10,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3744804[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:10,951 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:10,951 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,958 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:10,958 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,958 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:10,958 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:10,958 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:10,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:10,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,959 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:10,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:10,962 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:10,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:10,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:10,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:10,962 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:10,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:10,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:10,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:10,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:10,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:10,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:10,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:10,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:10,968 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:10,969 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:10,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:10,968 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:10,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,013 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=129, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/e117af3e49204d8f9ab679e6dfb50509 2023-04-28 08:55:11,069 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,072 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,075 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,075 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,075 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:11,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,076 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,076 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,076 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,076 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,076 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,076 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,077 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,077 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:11,077 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:11,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:11,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:11,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:11,078 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,079 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,079 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:11,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125ec218[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:11,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:11,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:11,081 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:11,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:11,081 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:11,082 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,082 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:11,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:11,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:11,085 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:11,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:11,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:11,085 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:11,085 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:11,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,087 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,088 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,089 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,091 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,091 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,091 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,091 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:11,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:11,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,092 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:11,092 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:11,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,092 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,192 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,203 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,203 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,203 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,203 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,203 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,203 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,204 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,204 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,206 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,206 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,206 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,206 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:11,207 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:11,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:11,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:11,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:11,208 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,208 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,208 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:11,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e2e3cc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:11,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:11,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,212 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:11,212 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:11,212 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:11,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:11,215 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,215 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:11,216 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,216 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:11,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:11,218 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:11,218 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:11,218 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:11,218 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:11,218 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:11,219 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,219 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,219 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,220 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,221 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:11,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:11,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,223 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,224 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:11,224 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:11,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:11,324 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,327 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,327 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,327 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,334 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,334 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,335 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:11,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:11,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:11,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:11,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,338 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,338 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,338 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:11,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5748f16f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:11,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:11,339 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,340 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:11,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:11,341 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:11,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,341 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:11,342 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,342 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:11,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:11,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:11,344 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:11,344 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:11,344 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:11,344 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:11,344 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:11,348 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,348 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,349 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,349 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,350 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,352 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:11,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,353 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:11,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:11,353 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,353 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:11,353 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,353 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:11,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672171351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:11,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672171353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/dc9a97d104a94afb9eb969a376f3ffcb 2023-04-28 08:55:11,453 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,456 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:11,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,459 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,460 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,460 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,460 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:11,460 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:11,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:11,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:11,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:11,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,461 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,462 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,462 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:11,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76dda7ea[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:11,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:11,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,465 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:11,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:11,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:11,465 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,465 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:11,466 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:11,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:11,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:11,469 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:11,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:11,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:11,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:11,469 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:11,470 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,470 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,471 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,472 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,474 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,474 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,474 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,474 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:11,474 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:11,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:11,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,474 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,475 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,475 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,475 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:11,475 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,475 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:11,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,475 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,575 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,578 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,581 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,581 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,581 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:11,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:11,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:11,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:11,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,584 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:11,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:11,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f7f52b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:11,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:11,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:11,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:11,587 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,587 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:11,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,588 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:11,589 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,589 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:11,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:11,591 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:11,591 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:11,591 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:11,591 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:11,591 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:11,596 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,596 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,596 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,597 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,604 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,604 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,605 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:11,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:11,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,605 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:11,605 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:11,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,705 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,709 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,709 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,709 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,709 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,709 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,709 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,711 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:11,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,712 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,712 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,713 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:11,713 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,713 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:11,713 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:11,713 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,713 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:11,713 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:11,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,714 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:11,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@435a983a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:11,716 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:11,716 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,718 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,718 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:11,718 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:11,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,718 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:11,718 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:11,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:11,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,719 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:11,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:11,722 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:11,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:11,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:11,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:11,722 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:11,723 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,723 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,724 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,724 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,725 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,727 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,727 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,727 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,727 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:11,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:11,727 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:11,728 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:11,728 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,828 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,831 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,832 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,832 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/dde1bcf9c66d4c19aef03db62013851d 2023-04-28 08:55:11,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/e117af3e49204d8f9ab679e6dfb50509 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e117af3e49204d8f9ab679e6dfb50509 2023-04-28 08:55:11,838 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,838 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,839 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,839 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:11,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,840 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:11,840 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:11,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:11,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:11,841 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:11,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,842 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,842 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:11,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5389cab7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:11,845 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:11,845 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e117af3e49204d8f9ab679e6dfb50509, entries=250, sequenceid=129, filesize=38.7 K 2023-04-28 08:55:11,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:11,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,848 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:11,848 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:11,848 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:11,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/dc9a97d104a94afb9eb969a376f3ffcb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dc9a97d104a94afb9eb969a376f3ffcb 2023-04-28 08:55:11,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:11,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:11,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:11,852 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:11,852 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:11,852 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:11,852 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:11,852 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:11,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dc9a97d104a94afb9eb969a376f3ffcb, entries=150, sequenceid=129, filesize=11.6 K 2023-04-28 08:55:11,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/dde1bcf9c66d4c19aef03db62013851d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dde1bcf9c66d4c19aef03db62013851d 2023-04-28 08:55:11,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:11,855 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,856 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:11,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,857 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:11,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:11,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:11,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:11,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:11,859 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,859 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:11,859 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:11,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:11,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:11,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dde1bcf9c66d4c19aef03db62013851d, entries=150, sequenceid=129, filesize=11.6 K 2023-04-28 08:55:11,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~240.80 KB/246576, currentSize=93.93 KB/96180 for 5fb523e858d6851c02bb0f82113765f7 in 1682ms, sequenceid=129, compaction requested=true 2023-04-28 08:55:11,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:11,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:11,862 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:11,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:11,862 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:11,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:11,863 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91260 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:11,863 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:11,863 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,863 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3520118f451949199b8a8f0ceb88f4aa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ae0050181c6a489fb636b3dd33a8602f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e117af3e49204d8f9ab679e6dfb50509] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=89.1 K 2023-04-28 08:55:11,863 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:11,863 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:11,863 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 3520118f451949199b8a8f0ceb88f4aa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682672106801 2023-04-28 08:55:11,863 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,863 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49dfbcde51a44d780947cf585aa6155, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49ec9d1b65e4efe9bb661604829165c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dc9a97d104a94afb9eb969a376f3ffcb] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=35.0 K 2023-04-28 08:55:11,864 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ae0050181c6a489fb636b3dd33a8602f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682672107947 2023-04-28 08:55:11,864 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c49dfbcde51a44d780947cf585aa6155, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682672106801 2023-04-28 08:55:11,864 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e117af3e49204d8f9ab679e6dfb50509, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682672110146 2023-04-28 08:55:11,864 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c49ec9d1b65e4efe9bb661604829165c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682672107947 2023-04-28 08:55:11,864 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dc9a97d104a94afb9eb969a376f3ffcb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682672110164 2023-04-28 08:55:11,872 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#324 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:11,876 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#323 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:11,882 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/90b722e5cc4a41d886c6ad080b4ffd41 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/90b722e5cc4a41d886c6ad080b4ffd41 2023-04-28 08:55:11,887 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into 90b722e5cc4a41d886c6ad080b4ffd41(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:11,887 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:11,887 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=13, startTime=1682672111862; duration=0sec 2023-04-28 08:55:11,887 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:11,887 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:11,888 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:11,888 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:11,888 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,889 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/aa2db71d3fc84126acc158fbca44b861, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3c2ed61ef2ef448e800e744164f29589, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dde1bcf9c66d4c19aef03db62013851d] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=35.0 K 2023-04-28 08:55:11,889 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting aa2db71d3fc84126acc158fbca44b861, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1682672106801 2023-04-28 08:55:11,889 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3c2ed61ef2ef448e800e744164f29589, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1682672107947 2023-04-28 08:55:11,890 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dde1bcf9c66d4c19aef03db62013851d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682672110164 2023-04-28 08:55:11,895 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#325 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:11,912 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/b24b402e1bcc462d95dd25f06c021cb6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b24b402e1bcc462d95dd25f06c021cb6 2023-04-28 08:55:11,914 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/c8df26c2f8424f5bb37136c6e3125755 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c8df26c2f8424f5bb37136c6e3125755 2023-04-28 08:55:11,920 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into b24b402e1bcc462d95dd25f06c021cb6(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:11,920 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:11,920 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=13, startTime=1682672111862; duration=0sec 2023-04-28 08:55:11,920 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into c8df26c2f8424f5bb37136c6e3125755(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:11,920 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:11,920 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:11,920 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=13, startTime=1682672111862; duration=0sec 2023-04-28 08:55:11,921 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:11,960 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:11,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:11,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:11,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:11,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:11,962 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:11,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:11,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:11,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,963 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:11,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:11,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:11,965 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:11,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:11,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:11,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:11,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:11,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:11,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:11,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:11,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:11,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:11,966 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:11,966 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:11,966 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-28 08:55:11,967 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:11,967 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:11,967 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:11,967 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:11,967 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:11,967 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:11,982 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428aedf8f2b5fc04d0293200a21fdd2cc16 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aedf8f2b5fc04d0293200a21fdd2cc16 2023-04-28 08:55:11,992 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/5ae8239fa4704646a636a46f2956a319 2023-04-28 08:55:12,009 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/a83662db67fc43e0a2d189e0f1a1d82b 2023-04-28 08:55:12,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:12,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:12,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672172392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672172392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,421 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/daca2ec0428448ee81c6c95d4b473b24 2023-04-28 08:55:12,425 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/5ae8239fa4704646a636a46f2956a319 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/5ae8239fa4704646a636a46f2956a319 2023-04-28 08:55:12,429 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/5ae8239fa4704646a636a46f2956a319, entries=150, sequenceid=149, filesize=25.3 K 2023-04-28 08:55:12,429 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/a83662db67fc43e0a2d189e0f1a1d82b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/a83662db67fc43e0a2d189e0f1a1d82b 2023-04-28 08:55:12,433 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/a83662db67fc43e0a2d189e0f1a1d82b, entries=150, sequenceid=149, filesize=11.8 K 2023-04-28 08:55:12,434 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/daca2ec0428448ee81c6c95d4b473b24 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/daca2ec0428448ee81c6c95d4b473b24 2023-04-28 08:55:12,437 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/daca2ec0428448ee81c6c95d4b473b24, entries=150, sequenceid=149, filesize=11.8 K 2023-04-28 08:55:12,438 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5fb523e858d6851c02bb0f82113765f7 in 472ms, sequenceid=149, compaction requested=false 2023-04-28 08:55:12,438 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:12,438 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:12,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:12,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:12,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:12,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:12,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4870c5db[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:12,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:12,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,444 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:12,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:12,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:12,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:12,445 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:12,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:12,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:12,448 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:12,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:12,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:12,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:12,449 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:12,450 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,450 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,450 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,455 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:12,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:12,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,456 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,456 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:12,456 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:12,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:55:12,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:12,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:12,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:12,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:12,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:12,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:12,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672172529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672172534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,556 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:12,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:12,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:12,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:12,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:12,559 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:12,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:12,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:12,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,560 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,561 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,562 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:12,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:12,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:12,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:12,562 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:12,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:12,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:12,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:12,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:12,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:12,564 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:12,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:12,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:12,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:12,565 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:12,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65032f53[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:12,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:12,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:12,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:12,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:12,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:12,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,570 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:12,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:12,573 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:12,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:12,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:12,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:12,573 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:12,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,575 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,578 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,578 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,578 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,578 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:12,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:12,578 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,579 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:12,579 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:12,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672172636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672172639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,679 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:12,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:12,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:12,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:12,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:12,682 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:12,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:12,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:12,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,684 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:12,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:12,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:12,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:12,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:12,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:12,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:12,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:12,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:12,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:12,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:12,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:12,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:12,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:12,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:12,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@728fe081[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:12,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:12,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,692 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:12,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:12,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:12,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:12,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:12,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,693 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:12,694 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:12,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:12,696 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:12,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:12,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:12,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:12,696 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:12,697 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,697 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,698 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:12,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:12,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:12,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,703 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:12,703 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:12,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,804 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:12,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:12,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:12,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:12,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:12,807 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:12,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:12,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:12,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,808 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,808 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,808 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:12,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:12,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:12,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:12,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:12,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:12,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:12,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:12,811 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,811 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,812 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,812 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:12,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fce5015[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:12,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:12,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,814 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:12,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:12,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:12,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:12,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:12,817 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,817 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:12,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:12,818 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:12,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:12,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:12,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:12,819 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:12,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:12,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:12,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:12,823 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,824 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,824 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:12,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,824 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:12,841 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672172840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672172842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,911 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304280107ff63578d497881ee0799ef06ba02 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280107ff63578d497881ee0799ef06ba02 2023-04-28 08:55:12,924 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:12,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:12,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:12,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:12,927 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:12,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:12,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:12,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,937 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,937 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,937 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,938 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:12,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:12,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:12,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:12,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:12,939 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,939 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:12,940 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:12,941 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:12,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21de1e2a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:12,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:12,943 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,944 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:12,944 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:12,944 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:12,944 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:12,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:12,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:12,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:12,948 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:12,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:12,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:12,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:12,948 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:12,950 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,950 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,950 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:12,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,953 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:12,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:12,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:12,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:12,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:12,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:12,955 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:12,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:12,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:12,955 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:12,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,055 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:13,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:13,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:13,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:13,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:13,059 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:13,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:13,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:13,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:13,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:13,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:13,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:13,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:13,062 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,062 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:13,062 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:13,062 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:13,062 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:13,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:13,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:13,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5bd4f5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:13,065 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:13,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,068 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:13,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:13,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:13,068 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,068 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:13,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:13,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,069 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:13,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:13,071 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:13,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:13,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:13,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:13,071 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:13,073 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,073 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,073 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,075 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:13,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:13,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,081 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:13,081 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:13,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:13,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672173142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672173148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672173149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,153 DEBUG [Thread-2275] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:13,181 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:13,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:13,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:13,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:13,184 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:13,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:13,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:13,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:13,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:13,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:13,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:13,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:13,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:13,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:13,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:13,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,190 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,191 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,191 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:13,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@505dac37[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:13,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:13,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,193 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,193 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:13,193 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:13,193 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:13,194 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,194 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:13,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:13,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:13,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:13,197 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:13,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:13,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:13,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:13,198 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:13,200 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,200 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,201 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,203 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,205 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:13,205 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,206 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:13,206 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:13,306 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:13,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:13,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:13,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:13,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:13,309 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:13,309 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:13,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:13,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,311 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,312 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:13,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:13,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:13,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:13,313 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,313 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:13,313 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:13,313 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:13,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:13,316 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/06cae32d8f0c4b8a9bc86fad597aadd3 2023-04-28 08:55:13,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:13,320 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:13,320 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0e10d1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:13,321 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:13,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:13,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:13,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:13,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:13,323 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:13,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:13,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:13,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:13,323 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:13,324 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:13,324 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,324 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:13,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,326 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,327 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,327 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/b654271ff12b41839ebab21a2a8b601c 2023-04-28 08:55:13,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:13,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:13,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,331 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:13,331 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:13,331 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:13,432 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:13,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:13,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:13,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:13,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:13,435 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:13,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:13,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:13,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,436 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,437 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:13,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:13,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:13,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:13,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:13,438 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,438 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:13,438 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:13,438 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:13,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:13,440 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:13,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf07814[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:13,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:13,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:13,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:13,443 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:13,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:13,445 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:13,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:13,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:13,447 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:13,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:13,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:13,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:13,448 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:13,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:13,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:13,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:13,454 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:13,454 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:13,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,554 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:13,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:13,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:13,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:13,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:13,557 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:13,557 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,557 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:13,557 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:13,558 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,558 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,559 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:13,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:13,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:13,560 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:13,560 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,560 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:13,560 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:13,560 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:13,560 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,560 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:13,561 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:13,561 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:13,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:13,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:13,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:13,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,563 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:13,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0feb03[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:13,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:13,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,565 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:13,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:13,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:13,566 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:13,567 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:13,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:13,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:13,569 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:13,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:13,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:13,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:13,569 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:13,571 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,571 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,571 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:13,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:13,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,576 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:13,576 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:13,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:13,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:13,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48378 deadline: 1682672173649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:13,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48424 deadline: 1682672173656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,676 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:13,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:13,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:13,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:13,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:13,680 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:13,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:13,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:13,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:13,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:13,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:13,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:13,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:13,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:13,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:13,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:13,684 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:13,684 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:13,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:13,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:13,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:13,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,687 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:13,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228ed0c1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:13,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:13,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:13,690 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:13,690 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:13,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:13,694 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:13,694 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,694 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,694 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:13,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:13,696 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:13,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:13,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:13,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:13,696 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:13,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:13,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:13,700 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:13,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:13,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:13,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:13,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:13,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:13,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:13,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,704 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:13,704 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:13,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:13,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:13,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/7ed5b2fcda9b4da0b91de9ba659eff1e 2023-04-28 08:55:13,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/06cae32d8f0c4b8a9bc86fad597aadd3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/06cae32d8f0c4b8a9bc86fad597aadd3 2023-04-28 08:55:13,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/06cae32d8f0c4b8a9bc86fad597aadd3, entries=200, sequenceid=165, filesize=32.1 K 2023-04-28 08:55:13,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/b654271ff12b41839ebab21a2a8b601c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b654271ff12b41839ebab21a2a8b601c 2023-04-28 08:55:13,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b654271ff12b41839ebab21a2a8b601c, entries=150, sequenceid=165, filesize=11.8 K 2023-04-28 08:55:13,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/7ed5b2fcda9b4da0b91de9ba659eff1e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7ed5b2fcda9b4da0b91de9ba659eff1e 2023-04-28 08:55:13,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7ed5b2fcda9b4da0b91de9ba659eff1e, entries=150, sequenceid=165, filesize=11.8 K 2023-04-28 08:55:13,758 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5fb523e858d6851c02bb0f82113765f7 in 1262ms, sequenceid=165, compaction requested=true 2023-04-28 08:55:13,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:13,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:13,758 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:13,759 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:13,758 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:13,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:13,759 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84768 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:13,759 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:13,759 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:13,759 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:13,760 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,760 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,760 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c8df26c2f8424f5bb37136c6e3125755, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/5ae8239fa4704646a636a46f2956a319, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/06cae32d8f0c4b8a9bc86fad597aadd3] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=82.8 K 2023-04-28 08:55:13,760 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/90b722e5cc4a41d886c6ad080b4ffd41, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/a83662db67fc43e0a2d189e0f1a1d82b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b654271ff12b41839ebab21a2a8b601c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=35.3 K 2023-04-28 08:55:13,760 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting c8df26c2f8424f5bb37136c6e3125755, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682672110164 2023-04-28 08:55:13,760 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 90b722e5cc4a41d886c6ad080b4ffd41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682672110164 2023-04-28 08:55:13,760 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 5ae8239fa4704646a636a46f2956a319, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672110213 2023-04-28 08:55:13,760 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a83662db67fc43e0a2d189e0f1a1d82b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672110213 2023-04-28 08:55:13,761 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 06cae32d8f0c4b8a9bc86fad597aadd3, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682672112369 2023-04-28 08:55:13,761 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b654271ff12b41839ebab21a2a8b601c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682672112369 2023-04-28 08:55:13,769 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:13,779 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#333 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:13,804 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:13,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:13,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:13,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:13,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:13,807 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:13,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:13,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:13,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,808 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:13,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:13,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:13,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:13,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:13,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:13,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:13,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:13,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:13,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:13,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:13,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:13,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:13,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:13,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:13,810 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-28 08:55:13,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:13,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:13,810 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:13,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:13,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:13,811 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:14,188 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/fc0bff06f9ba4f4b86b46858a182dbe2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fc0bff06f9ba4f4b86b46858a182dbe2 2023-04-28 08:55:14,193 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into fc0bff06f9ba4f4b86b46858a182dbe2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:14,193 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:14,193 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=13, startTime=1682672113758; duration=0sec 2023-04-28 08:55:14,193 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:14,193 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:14,194 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:14,194 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:14,194 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:14,194 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b24b402e1bcc462d95dd25f06c021cb6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/daca2ec0428448ee81c6c95d4b473b24, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7ed5b2fcda9b4da0b91de9ba659eff1e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=35.3 K 2023-04-28 08:55:14,195 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b24b402e1bcc462d95dd25f06c021cb6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1682672110164 2023-04-28 08:55:14,195 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting daca2ec0428448ee81c6c95d4b473b24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672110213 2023-04-28 08:55:14,196 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7ed5b2fcda9b4da0b91de9ba659eff1e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682672112369 2023-04-28 08:55:14,208 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#335 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:14,229 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/3db7290a37ff46b1b19c9c7827095708 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3db7290a37ff46b1b19c9c7827095708 2023-04-28 08:55:14,236 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into 3db7290a37ff46b1b19c9c7827095708(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:14,236 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:14,236 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=13, startTime=1682672113758; duration=0sec 2023-04-28 08:55:14,236 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:14,237 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ac90ee39eda24e9ca166b0f8a893b76e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ac90ee39eda24e9ca166b0f8a893b76e 2023-04-28 08:55:14,244 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/9c5c95ff5b09406f97350b2ff3a0fd13 2023-04-28 08:55:14,640 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/97cc52fe20dc4787bf3a5dede4080b1c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/97cc52fe20dc4787bf3a5dede4080b1c 2023-04-28 08:55:14,644 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into 97cc52fe20dc4787bf3a5dede4080b1c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:14,644 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:14,644 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=13, startTime=1682672113759; duration=0sec 2023-04-28 08:55:14,645 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:14,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:14,659 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/bb4927774a624cd2bf38ec1b1c262260 2023-04-28 08:55:14,669 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/94fc842d8c18429fa5ffbbf4be04cf86 2023-04-28 08:55:14,672 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/9c5c95ff5b09406f97350b2ff3a0fd13 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/9c5c95ff5b09406f97350b2ff3a0fd13 2023-04-28 08:55:14,676 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/9c5c95ff5b09406f97350b2ff3a0fd13, entries=150, sequenceid=181, filesize=25.3 K 2023-04-28 08:55:14,676 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/bb4927774a624cd2bf38ec1b1c262260 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb4927774a624cd2bf38ec1b1c262260 2023-04-28 08:55:14,680 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb4927774a624cd2bf38ec1b1c262260, entries=150, sequenceid=181, filesize=11.8 K 2023-04-28 08:55:14,681 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/94fc842d8c18429fa5ffbbf4be04cf86 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/94fc842d8c18429fa5ffbbf4be04cf86 2023-04-28 08:55:14,684 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/94fc842d8c18429fa5ffbbf4be04cf86, entries=150, sequenceid=181, filesize=11.8 K 2023-04-28 08:55:14,685 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=46.96 KB/48090 for 5fb523e858d6851c02bb0f82113765f7 in 875ms, sequenceid=181, compaction requested=false 2023-04-28 08:55:14,685 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:14,685 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:14,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:14,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:14,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:14,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:55:14,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:14,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:14,685 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:14,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:14,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:14,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:14,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:14,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:14,688 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,689 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:14,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f3bc4d9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:14,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:14,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,691 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:14,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:14,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:14,697 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,697 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:14,698 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:14,698 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:14,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,698 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:14,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:14,701 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:14,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:14,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:14,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:14,701 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:14,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:14,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:14,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,704 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,706 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,706 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,706 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:14,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:14,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:14,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:14,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:14,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:14,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:14,707 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:14,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:14,707 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:14,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672114728, max timestamp=1682672114741Num uniques 859; 2023-04-28 08:55:14,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672114728, max timestamp=1682672114741Num uniques 859; 2023-04-28 08:55:14,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:14,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:14,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672114728, max timestamp=1682672114741Num uniques 859; 2023-04-28 08:55:14,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:14,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:14,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672174790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:14,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672174791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,807 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:14,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:14,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:14,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:14,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:14,810 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:14,810 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:14,810 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:14,810 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:14,811 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,811 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,812 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:14,812 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:14,812 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,812 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:14,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:14,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:14,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:14,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:14,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:14,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:14,813 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:14,813 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:14,814 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:14,814 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:14,814 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:14,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:14,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:14,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:14,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:14,815 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,815 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,815 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:14,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@142ff2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:14,817 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:14,817 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,818 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,818 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:14,818 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:14,818 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:14,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:14,821 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:14,821 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:14,821 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,821 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:14,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:14,823 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:14,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:14,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:14,824 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:14,824 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:14,825 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,825 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,825 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:14,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:14,825 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,827 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:14,829 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:14,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:14,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:14,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:14,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:14,829 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:14,830 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:14,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:14,830 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:14,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:14,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672174897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672174898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:14,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672174925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,927 DEBUG [Thread-2277] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:14,930 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:14,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:14,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:14,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672174928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:14,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:14,933 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:14,933 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:14,933 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:14,933 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:14,934 DEBUG [Thread-2273] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8191 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:14,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,936 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,936 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:14,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:14,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,936 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,936 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:14,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:14,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:14,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:14,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:14,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:14,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:14,937 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:14,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:14,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:14,938 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:14,938 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:14,938 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:14,938 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:14,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:14,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:14,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:14,938 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:14,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,940 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:14,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60a38b00[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:14,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:14,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,943 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,943 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:14,944 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:14,944 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:14,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,944 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:14,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:14,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:14,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:14,947 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:14,947 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:14,947 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:14,947 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:14,947 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:14,952 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,952 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:14,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,952 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:14,953 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:14,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:14,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,955 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:14,956 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:14,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:14,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:14,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:14,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:14,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:14,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:14,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:14,957 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:14,957 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:15,057 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:15,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:15,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:15,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:15,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:15,060 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:15,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:15,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:15,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,062 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,062 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,062 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,063 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:15,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:15,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:15,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:15,064 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,064 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:15,064 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:15,064 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:15,064 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:15,065 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,065 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:15,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ff470d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:15,067 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:15,067 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,070 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:15,070 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:15,070 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:15,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:15,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,071 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:15,071 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,071 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:15,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:15,074 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:15,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:15,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:15,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:15,074 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:15,077 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,077 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,077 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,082 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,083 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,085 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,085 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:15,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:15,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:15,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,086 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:15,086 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:15,102 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428527beb3916094560b02ba29377f9ada1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428527beb3916094560b02ba29377f9ada1 2023-04-28 08:55:15,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:15,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672175104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,110 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=195, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/473d4b7e01cd42ba8e69a8b505ed22b5 2023-04-28 08:55:15,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:15,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672175107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/719d5af07238429eae98a06c7bf444df 2023-04-28 08:55:15,187 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:15,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:15,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:15,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:15,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:15,190 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:15,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:15,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:15,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,193 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,193 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:15,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:15,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:15,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:15,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:15,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:15,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:15,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:15,194 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,194 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:15,194 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:15,194 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:15,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:15,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:15,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:15,196 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,196 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,197 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,197 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:15,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75234047[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:15,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:15,198 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,199 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,199 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:15,199 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:15,199 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:15,200 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,200 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:15,201 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:15,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,201 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:15,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:15,204 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:15,204 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:15,204 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:15,204 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:15,204 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:15,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,207 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,207 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,210 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,210 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,210 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,210 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,210 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:15,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:15,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,211 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:15,211 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:15,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,311 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:15,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:15,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:15,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:15,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:15,314 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:15,314 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,314 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:15,314 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:15,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,316 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:15,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:15,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:15,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:15,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:15,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:15,317 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,318 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:15,318 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:15,318 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:15,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:15,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:15,318 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:15,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:15,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e99aae[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:15,321 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:15,321 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,323 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,323 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:15,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:15,323 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:15,323 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:15,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:15,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,325 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:15,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:15,327 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:15,327 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:15,327 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:15,327 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:15,327 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:15,329 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,329 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,329 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,329 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:15,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:15,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,338 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,338 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:15,338 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:15,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:15,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672175410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:15,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672175413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,438 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:15,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:15,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:15,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:15,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:15,441 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:15,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:15,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:15,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:15,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:15,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:15,445 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,445 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:15,445 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:15,445 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:15,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:15,447 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:15,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28eef0b8[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:15,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:15,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:15,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:15,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:15,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:15,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:15,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:15,457 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:15,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:15,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:15,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:15,457 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:15,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:15,459 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:15,460 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,462 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:15,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:15,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:15,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:15,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,462 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:15,462 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:15,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:15,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:15,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:15,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=195 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/173d20a077654d859e104948f683a3a1 2023-04-28 08:55:15,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/473d4b7e01cd42ba8e69a8b505ed22b5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/473d4b7e01cd42ba8e69a8b505ed22b5 2023-04-28 08:55:15,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/473d4b7e01cd42ba8e69a8b505ed22b5, entries=200, sequenceid=195, filesize=32.1 K 2023-04-28 08:55:15,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/719d5af07238429eae98a06c7bf444df as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/719d5af07238429eae98a06c7bf444df 2023-04-28 08:55:15,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/719d5af07238429eae98a06c7bf444df, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 08:55:15,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/173d20a077654d859e104948f683a3a1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/173d20a077654d859e104948f683a3a1 2023-04-28 08:55:15,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/173d20a077654d859e104948f683a3a1, entries=150, sequenceid=195, filesize=11.8 K 2023-04-28 08:55:15,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5fb523e858d6851c02bb0f82113765f7 in 864ms, sequenceid=195, compaction requested=true 2023-04-28 08:55:15,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:15,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:15,549 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:15,549 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:15,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:15,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:15,550 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:15,550 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:15,550 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:15,550 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:15,550 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,550 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,550 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3db7290a37ff46b1b19c9c7827095708, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/9c5c95ff5b09406f97350b2ff3a0fd13, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/473d4b7e01cd42ba8e69a8b505ed22b5] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=83.0 K 2023-04-28 08:55:15,550 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fc0bff06f9ba4f4b86b46858a182dbe2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb4927774a624cd2bf38ec1b1c262260, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/719d5af07238429eae98a06c7bf444df] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=35.6 K 2023-04-28 08:55:15,551 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 3db7290a37ff46b1b19c9c7827095708, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682672112369 2023-04-28 08:55:15,551 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting fc0bff06f9ba4f4b86b46858a182dbe2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682672112369 2023-04-28 08:55:15,551 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9c5c95ff5b09406f97350b2ff3a0fd13, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682672112514 2023-04-28 08:55:15,551 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bb4927774a624cd2bf38ec1b1c262260, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682672112514 2023-04-28 08:55:15,551 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 473d4b7e01cd42ba8e69a8b505ed22b5, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672114667 2023-04-28 08:55:15,552 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 719d5af07238429eae98a06c7bf444df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672114667 2023-04-28 08:55:15,559 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:15,561 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#341 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:15,562 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:15,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:15,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:15,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:15,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:15,571 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:15,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:15,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:15,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,572 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:15,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:15,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:15,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:15,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:15,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:15,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:15,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:15,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:15,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:15,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:15,576 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:15,576 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:15,576 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:15,592 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d2c827b18b3641f59dff38cee3fa6050 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d2c827b18b3641f59dff38cee3fa6050 2023-04-28 08:55:15,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:15,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:15,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:15,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672175944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:15,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672175948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:15,968 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/dde6d6d534a640f9a81edce8704eabe7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dde6d6d534a640f9a81edce8704eabe7 2023-04-28 08:55:15,972 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into dde6d6d534a640f9a81edce8704eabe7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:15,972 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:15,972 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=13, startTime=1682672115549; duration=0sec 2023-04-28 08:55:15,972 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:15,973 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:15,974 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:15,974 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:15,975 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:15,975 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/97cc52fe20dc4787bf3a5dede4080b1c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/94fc842d8c18429fa5ffbbf4be04cf86, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/173d20a077654d859e104948f683a3a1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=35.6 K 2023-04-28 08:55:15,975 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 97cc52fe20dc4787bf3a5dede4080b1c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1682672112369 2023-04-28 08:55:15,975 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 94fc842d8c18429fa5ffbbf4be04cf86, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1682672112514 2023-04-28 08:55:15,976 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 173d20a077654d859e104948f683a3a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672114667 2023-04-28 08:55:15,981 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#344 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:16,001 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/83d8ceabf37149a4be2ae50f89f9b8dd 2023-04-28 08:55:16,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672176056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672176057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672176262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672176263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,387 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/cb49d5944ba34b1e943e5932510e8498 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/cb49d5944ba34b1e943e5932510e8498 2023-04-28 08:55:16,388 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/4be180ed2cce4a6bb07a1a9e89b6e1de as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/4be180ed2cce4a6bb07a1a9e89b6e1de 2023-04-28 08:55:16,391 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into cb49d5944ba34b1e943e5932510e8498(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:16,391 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:16,391 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=13, startTime=1682672115549; duration=0sec 2023-04-28 08:55:16,392 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:16,393 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into 4be180ed2cce4a6bb07a1a9e89b6e1de(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:16,394 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:16,394 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=13, startTime=1682672115549; duration=0sec 2023-04-28 08:55:16,394 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:16,410 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/9476861abecb4106bc08acfbcbf5099c 2023-04-28 08:55:16,428 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/b1c67bde72d84612a9d9424ce2cd61d3 2023-04-28 08:55:16,432 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/83d8ceabf37149a4be2ae50f89f9b8dd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/83d8ceabf37149a4be2ae50f89f9b8dd 2023-04-28 08:55:16,436 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/83d8ceabf37149a4be2ae50f89f9b8dd, entries=150, sequenceid=231, filesize=25.3 K 2023-04-28 08:55:16,437 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/9476861abecb4106bc08acfbcbf5099c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/9476861abecb4106bc08acfbcbf5099c 2023-04-28 08:55:16,440 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/9476861abecb4106bc08acfbcbf5099c, entries=150, sequenceid=231, filesize=11.8 K 2023-04-28 08:55:16,441 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/b1c67bde72d84612a9d9424ce2cd61d3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b1c67bde72d84612a9d9424ce2cd61d3 2023-04-28 08:55:16,444 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b1c67bde72d84612a9d9424ce2cd61d3, entries=150, sequenceid=231, filesize=11.8 K 2023-04-28 08:55:16,445 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 5fb523e858d6851c02bb0f82113765f7 in 869ms, sequenceid=231, compaction requested=false 2023-04-28 08:55:16,445 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:16,445 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:16,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:16,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:16,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:16,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:16,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:16,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:16,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:16,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:16,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:16,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66671d87[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:16,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:16,450 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:16,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:16,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:16,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:16,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:16,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:16,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:16,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:16,453 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:16,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:16,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:16,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:16,453 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:16,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:16,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:16,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:16,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:16,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:16,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:16,461 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:16,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:16,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:16,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:16,461 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:16,461 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:16,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:16,461 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,561 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:16,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:16,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:16,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:16,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:16,564 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:16,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:16,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:16,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:16,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:16,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:16,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:16,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:16,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:16,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:16,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:16,567 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:16,567 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:16,567 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:16,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:16,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:16,568 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:16,568 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:16,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:16,569 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:16,588 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428b5c94a3566cb40139f53aef90766e9a5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5c94a3566cb40139f53aef90766e9a5 2023-04-28 08:55:16,594 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/7a8c954848b84a19a8f3e56ea88bf222 2023-04-28 08:55:16,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672176616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672116605, max timestamp=1682672116618Num uniques 859; 2023-04-28 08:55:16,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:16,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672116605, max timestamp=1682672116618Num uniques 859; 2023-04-28 08:55:16,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:16,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672116605, max timestamp=1682672116618Num uniques 859; 2023-04-28 08:55:16,620 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:16,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672176722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672176732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,837 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672176833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:16,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:16,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672176840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,013 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/d3d2f6873dfa4c25a318533b546bce4a 2023-04-28 08:55:17,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:17,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672177039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:17,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672177047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:17,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672177343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:17,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672177351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,423 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/ffb6826b5daa4135a981d06bd9d4a92f 2023-04-28 08:55:17,427 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/7a8c954848b84a19a8f3e56ea88bf222 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7a8c954848b84a19a8f3e56ea88bf222 2023-04-28 08:55:17,432 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7a8c954848b84a19a8f3e56ea88bf222, entries=150, sequenceid=245, filesize=25.3 K 2023-04-28 08:55:17,434 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/d3d2f6873dfa4c25a318533b546bce4a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/d3d2f6873dfa4c25a318533b546bce4a 2023-04-28 08:55:17,438 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/d3d2f6873dfa4c25a318533b546bce4a, entries=150, sequenceid=245, filesize=11.8 K 2023-04-28 08:55:17,439 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/ffb6826b5daa4135a981d06bd9d4a92f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/ffb6826b5daa4135a981d06bd9d4a92f 2023-04-28 08:55:17,443 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/ffb6826b5daa4135a981d06bd9d4a92f, entries=150, sequenceid=245, filesize=11.8 K 2023-04-28 08:55:17,443 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5fb523e858d6851c02bb0f82113765f7 in 875ms, sequenceid=245, compaction requested=true 2023-04-28 08:55:17,443 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:17,444 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:17,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:17,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:17,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:17,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:17,445 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,445 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:17,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:17,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:17,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:17,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9294874[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:17,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:17,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:17,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:17,448 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:17,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:17,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:17,451 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:17,451 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:17,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:17,453 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:17,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:17,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:17,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:17,453 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:17,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:17,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:17,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:17,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:17,455 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:17,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,462 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:17,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:17,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:17,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:17,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:17,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:17,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:17,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:17,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:17,463 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:17,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:17,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,463 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:17,563 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:17,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:17,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:17,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:17,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:17,566 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:17,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:17,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:17,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:17,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:17,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:17,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:17,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:17,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:17,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:17,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:17,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:17,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:17,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:17,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:17,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:17,571 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:17,571 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:17,571 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-28 08:55:17,572 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:17,572 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:17,572 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:17,572 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:17,572 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:17,572 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:17,586 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e95fefb7cb4d401b8ead9ce30398b0f9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e95fefb7cb4d401b8ead9ce30398b0f9 2023-04-28 08:55:17,591 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=281, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/d5e07cca93914a3f98390a90a7716082 2023-04-28 08:55:17,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:17,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:17,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:17,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672177890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:17,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:17,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672177892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672177998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672177998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,004 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/56d52410ba1e44eda6283d55c04be309 2023-04-28 08:55:18,026 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/96213c5595b94cf6aa81c32f8e547f8a 2023-04-28 08:55:18,034 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/d5e07cca93914a3f98390a90a7716082 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/d5e07cca93914a3f98390a90a7716082 2023-04-28 08:55:18,037 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/d5e07cca93914a3f98390a90a7716082, entries=150, sequenceid=281, filesize=25.4 K 2023-04-28 08:55:18,038 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/56d52410ba1e44eda6283d55c04be309 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/56d52410ba1e44eda6283d55c04be309 2023-04-28 08:55:18,043 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/56d52410ba1e44eda6283d55c04be309, entries=150, sequenceid=281, filesize=11.9 K 2023-04-28 08:55:18,044 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/96213c5595b94cf6aa81c32f8e547f8a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/96213c5595b94cf6aa81c32f8e547f8a 2023-04-28 08:55:18,049 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/96213c5595b94cf6aa81c32f8e547f8a, entries=150, sequenceid=281, filesize=11.9 K 2023-04-28 08:55:18,049 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 5fb523e858d6851c02bb0f82113765f7 in 478ms, sequenceid=281, compaction requested=true 2023-04-28 08:55:18,049 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:18,049 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:18,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:18,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:18,050 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:18,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:18,052 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,052 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,053 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@118d9f15[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:18,055 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:18,055 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,056 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:18,056 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:18,056 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:18,057 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,057 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:18,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:18,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,058 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:18,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:18,060 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:18,060 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:18,060 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:18,060 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:18,060 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:18,063 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,063 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,065 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,067 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,067 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:18,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:18,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:18,068 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:18,068 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:18,168 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:18,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:18,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:18,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:18,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:18,171 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:18,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:18,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:18,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:18,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:18,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:18,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:18,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:18,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:18,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:18,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:18,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:18,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:18,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:18,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:18,174 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:55:18,174 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:18,175 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:18,175 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:18,175 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:18,175 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:18,175 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:18,185 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304289843ef6c2a7945fa978e09200258c8ab to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289843ef6c2a7945fa978e09200258c8ab 2023-04-28 08:55:18,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:18,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:18,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672118241, max timestamp=1682672118276Num uniques 859; 2023-04-28 08:55:18,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672118241, max timestamp=1682672118276Num uniques 859; 2023-04-28 08:55:18,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:18,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672118241, max timestamp=1682672118276Num uniques 859; 2023-04-28 08:55:18,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:18,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:18,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672178338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672178339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672178446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672178447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,591 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=292, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/c606c00f288f4c95b0867131f150c403 2023-04-28 08:55:18,601 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/6a3b7b808ddd44bd98e593ef77fe56b5 2023-04-28 08:55:18,609 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/3d028c5059c84f7e96e68fdc723a4acf 2023-04-28 08:55:18,613 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/c606c00f288f4c95b0867131f150c403 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c606c00f288f4c95b0867131f150c403 2023-04-28 08:55:18,616 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c606c00f288f4c95b0867131f150c403, entries=150, sequenceid=292, filesize=25.4 K 2023-04-28 08:55:18,618 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/6a3b7b808ddd44bd98e593ef77fe56b5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/6a3b7b808ddd44bd98e593ef77fe56b5 2023-04-28 08:55:18,622 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/6a3b7b808ddd44bd98e593ef77fe56b5, entries=150, sequenceid=292, filesize=11.9 K 2023-04-28 08:55:18,623 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/3d028c5059c84f7e96e68fdc723a4acf as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3d028c5059c84f7e96e68fdc723a4acf 2023-04-28 08:55:18,626 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3d028c5059c84f7e96e68fdc723a4acf, entries=150, sequenceid=292, filesize=11.9 K 2023-04-28 08:55:18,627 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5fb523e858d6851c02bb0f82113765f7 in 453ms, sequenceid=292, compaction requested=true 2023-04-28 08:55:18,627 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:18,627 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:18,627 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:18,627 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:18,627 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,627 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:18,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:18,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,629 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,629 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,629 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:18,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb9d6ac[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:18,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:18,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,633 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:18,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:18,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:18,633 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,633 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:18,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:18,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:18,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:18,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:18,636 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:18,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:18,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:18,636 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:18,636 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:18,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,638 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:18,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:18,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:18,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:18,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:18,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:18,642 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:18,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,642 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:18,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:18,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-28 08:55:18,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:18,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:18,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:18,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:18,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:18,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:18,671 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428f7f213d292b44cf6b90ddc1dc403b90d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f7f213d292b44cf6b90ddc1dc403b90d 2023-04-28 08:55:18,711 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672178695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672178696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,742 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:18,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:18,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:18,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:18,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:18,749 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:18,749 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:18,749 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:18,749 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:18,749 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,749 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,751 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:18,751 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:18,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:18,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:18,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:18,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:18,753 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:18,753 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:18,753 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:18,753 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:18,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:18,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:18,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106c61d9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:18,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:18,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,757 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:18,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:18,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:18,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:18,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:18,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:18,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:18,761 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:18,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:18,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:18,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:18,761 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:18,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:18,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:18,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:18,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:18,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:18,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,768 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:18,768 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:18,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672178813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:18,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672178813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,868 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:18,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:18,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:18,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:18,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:18,872 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:18,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:18,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:18,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:18,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,873 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,873 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:18,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:18,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:18,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:18,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:18,875 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:18,875 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:18,875 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:18,875 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:18,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:18,880 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,880 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,880 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,880 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:18,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5345f148[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:18,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:18,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,883 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:18,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:18,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:18,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:18,884 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:18,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:18,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:18,887 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:18,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:18,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:18,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:18,887 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:18,888 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,888 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:18,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,888 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:18,889 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:18,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:18,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:18,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:18,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:18,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:18,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:18,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:18,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:18,895 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:18,895 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:18,995 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:18,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:18,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:18,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:18,998 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:18,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:18,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:18,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:18,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:18,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,001 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,001 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,002 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,002 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,003 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,003 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,003 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ecda902[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,005 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,006 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,006 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,006 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,008 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,010 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,010 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,012 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,012 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,012 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,012 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,015 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:19,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,016 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,016 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:19,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672179021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:19,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672179022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,076 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/60ff54f91cb24d599418b7d9d9e07077 2023-04-28 08:55:19,116 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:19,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:19,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:19,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:19,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:19,120 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:19,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:19,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:19,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,121 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,121 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,122 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,125 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,132 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,132 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,133 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec116c1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,133 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,134 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,135 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,135 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,135 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,140 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,140 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,141 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,141 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,141 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,142 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,143 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,148 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,148 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:19,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,248 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:19,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:19,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:19,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:19,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:19,252 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:19,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:19,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:19,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,254 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,255 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,255 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,255 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,255 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,255 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,256 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,256 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,257 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d50d869[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,260 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,260 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,267 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,267 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,267 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,268 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,268 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,269 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:19,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,273 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,273 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,272 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:19,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672179327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:19,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672179327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,373 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:19,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:19,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:19,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:19,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:19,376 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:19,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:19,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:19,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,378 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,382 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,383 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,383 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2163bd3a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,384 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,385 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,386 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,386 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,389 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,389 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,392 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,396 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,396 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,396 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,486 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/ddaeadf638684f438510e99dea493b2e 2023-04-28 08:55:19,496 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:19,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:19,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:19,499 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:19,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:19,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:19,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,501 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,502 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,502 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,502 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,503 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,504 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,504 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,506 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,506 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a585ad[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,508 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,508 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,508 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,508 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,510 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,513 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,513 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,513 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,514 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,514 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,514 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,516 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,518 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,518 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,519 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,519 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,619 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:19,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:19,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:19,625 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:19,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:19,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:19,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,625 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,626 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,627 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,627 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,628 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,628 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,628 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,628 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,628 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,630 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,631 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44f4e026[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,632 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,632 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,633 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,633 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,633 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,635 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,635 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,635 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,637 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,637 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,642 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,643 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,646 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,646 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,646 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,747 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:19,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:19,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:19,750 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:19,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:19,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:19,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,750 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,752 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,754 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,754 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,754 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,754 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f3ceb41[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,760 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,760 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,761 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,762 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,768 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,769 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,769 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,771 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:19,782 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,783 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,782 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,783 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:19,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672179834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:19,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672179834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,883 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:19,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:19,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:19,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:19,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:19,886 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:19,886 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,886 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:19,886 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:19,886 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,886 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,888 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:19,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:19,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:19,889 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,889 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:19,889 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:19,889 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:19,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:19,891 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@238a0ff1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:19,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:19,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:19,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:19,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:19,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:19,896 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,896 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:19,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:19,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/d057df3d14664778a4a821e4f13aba7a 2023-04-28 08:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:19,899 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:19,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:19,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:19,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:19,899 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:19,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/60ff54f91cb24d599418b7d9d9e07077 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/60ff54f91cb24d599418b7d9d9e07077 2023-04-28 08:55:19,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:19,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:19,902 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,906 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/60ff54f91cb24d599418b7d9d9e07077, entries=200, sequenceid=329, filesize=32.3 K 2023-04-28 08:55:19,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/ddaeadf638684f438510e99dea493b2e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ddaeadf638684f438510e99dea493b2e 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:19,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:19,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:19,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:19,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:19,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:19,908 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:19,908 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:19,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:19,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:19,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:19,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:19,910 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ddaeadf638684f438510e99dea493b2e, entries=150, sequenceid=329, filesize=11.9 K 2023-04-28 08:55:19,911 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/d057df3d14664778a4a821e4f13aba7a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/d057df3d14664778a4a821e4f13aba7a 2023-04-28 08:55:19,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/d057df3d14664778a4a821e4f13aba7a, entries=150, sequenceid=329, filesize=11.9 K 2023-04-28 08:55:19,915 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 5fb523e858d6851c02bb0f82113765f7 in 1257ms, sequenceid=329, compaction requested=true 2023-04-28 08:55:19,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:19,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:19,915 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:55:19,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:19,915 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:55:19,915 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:19,921 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73110 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:55:19,921 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163310 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:55:19,921 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:19,921 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:19,921 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,921 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,921 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dde6d6d534a640f9a81edce8704eabe7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/9476861abecb4106bc08acfbcbf5099c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/d3d2f6873dfa4c25a318533b546bce4a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/56d52410ba1e44eda6283d55c04be309, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/6a3b7b808ddd44bd98e593ef77fe56b5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ddaeadf638684f438510e99dea493b2e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=71.4 K 2023-04-28 08:55:19,921 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/cb49d5944ba34b1e943e5932510e8498, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/83d8ceabf37149a4be2ae50f89f9b8dd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7a8c954848b84a19a8f3e56ea88bf222, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/d5e07cca93914a3f98390a90a7716082, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c606c00f288f4c95b0867131f150c403, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/60ff54f91cb24d599418b7d9d9e07077] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=159.5 K 2023-04-28 08:55:19,922 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dde6d6d534a640f9a81edce8704eabe7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672114667 2023-04-28 08:55:19,922 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting cb49d5944ba34b1e943e5932510e8498, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672114667 2023-04-28 08:55:19,922 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9476861abecb4106bc08acfbcbf5099c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682672114782 2023-04-28 08:55:19,922 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 83d8ceabf37149a4be2ae50f89f9b8dd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682672114782 2023-04-28 08:55:19,922 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d3d2f6873dfa4c25a318533b546bce4a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682672115937 2023-04-28 08:55:19,923 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7a8c954848b84a19a8f3e56ea88bf222, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682672115937 2023-04-28 08:55:19,923 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 56d52410ba1e44eda6283d55c04be309, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682672116664 2023-04-28 08:55:19,923 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting d5e07cca93914a3f98390a90a7716082, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682672116664 2023-04-28 08:55:19,923 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6a3b7b808ddd44bd98e593ef77fe56b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682672117878 2023-04-28 08:55:19,923 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting c606c00f288f4c95b0867131f150c403, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682672117878 2023-04-28 08:55:19,923 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ddaeadf638684f438510e99dea493b2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682672118322 2023-04-28 08:55:19,924 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 60ff54f91cb24d599418b7d9d9e07077, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682672118322 2023-04-28 08:55:19,940 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#360 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:19,949 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#359 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:19,970 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/b99196fcb9804b008d19077527a27c03 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b99196fcb9804b008d19077527a27c03 2023-04-28 08:55:19,974 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into b99196fcb9804b008d19077527a27c03(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:19,974 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:19,975 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=10, startTime=1682672119915; duration=0sec 2023-04-28 08:55:19,975 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:19,975 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:55:19,977 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73110 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:55:19,978 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:19,978 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:19,978 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/4be180ed2cce4a6bb07a1a9e89b6e1de, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b1c67bde72d84612a9d9424ce2cd61d3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/ffb6826b5daa4135a981d06bd9d4a92f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/96213c5595b94cf6aa81c32f8e547f8a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3d028c5059c84f7e96e68fdc723a4acf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/d057df3d14664778a4a821e4f13aba7a] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=71.4 K 2023-04-28 08:55:19,978 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4be180ed2cce4a6bb07a1a9e89b6e1de, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=195, earliestPutTs=1682672114667 2023-04-28 08:55:19,979 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b1c67bde72d84612a9d9424ce2cd61d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1682672114782 2023-04-28 08:55:19,979 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ffb6826b5daa4135a981d06bd9d4a92f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1682672115937 2023-04-28 08:55:19,979 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 96213c5595b94cf6aa81c32f8e547f8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1682672116664 2023-04-28 08:55:19,980 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3d028c5059c84f7e96e68fdc723a4acf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1682672117878 2023-04-28 08:55:19,980 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d057df3d14664778a4a821e4f13aba7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682672118322 2023-04-28 08:55:19,988 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#361 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:20,003 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/995c43be65e543119f8f10f2cc9aa2e6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/995c43be65e543119f8f10f2cc9aa2e6 2023-04-28 08:55:20,008 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:20,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:20,008 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into 995c43be65e543119f8f10f2cc9aa2e6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:20,008 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:20,009 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=10, startTime=1682672119915; duration=0sec 2023-04-28 08:55:20,009 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:20,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:20,011 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:20,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:20,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:20,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:20,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:20,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:20,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:20,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:20,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:20,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:20,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:20,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:20,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:20,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:20,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:20,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:20,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:20,023 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:20,023 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:20,390 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/e8a3a1901a564a5da1ceb18d6a4c2360 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e8a3a1901a564a5da1ceb18d6a4c2360 2023-04-28 08:55:20,394 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into e8a3a1901a564a5da1ceb18d6a4c2360(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:20,394 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:20,394 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=10, startTime=1682672119915; duration=0sec 2023-04-28 08:55:20,395 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:20,444 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a05e6b3fdddb4eb686c4606dc5f09338 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a05e6b3fdddb4eb686c4606dc5f09338 2023-04-28 08:55:20,450 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=341, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/7b2427f2e33a4f2c8121b8b11cd6e556 2023-04-28 08:55:20,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:20,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:20,859 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/ee8bac3990b84953ade0e59a39465667 2023-04-28 08:55:20,868 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/e245ac5e4c8d44f2bc7016ab1c1951e0 2023-04-28 08:55:20,872 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/7b2427f2e33a4f2c8121b8b11cd6e556 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7b2427f2e33a4f2c8121b8b11cd6e556 2023-04-28 08:55:20,876 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7b2427f2e33a4f2c8121b8b11cd6e556, entries=150, sequenceid=341, filesize=25.4 K 2023-04-28 08:55:20,876 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/ee8bac3990b84953ade0e59a39465667 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ee8bac3990b84953ade0e59a39465667 2023-04-28 08:55:20,880 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ee8bac3990b84953ade0e59a39465667, entries=150, sequenceid=341, filesize=11.9 K 2023-04-28 08:55:20,881 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/e245ac5e4c8d44f2bc7016ab1c1951e0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/e245ac5e4c8d44f2bc7016ab1c1951e0 2023-04-28 08:55:20,884 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/e245ac5e4c8d44f2bc7016ab1c1951e0, entries=150, sequenceid=341, filesize=11.9 K 2023-04-28 08:55:20,885 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=100.63 KB/103050 for 5fb523e858d6851c02bb0f82113765f7 in 862ms, sequenceid=341, compaction requested=false 2023-04-28 08:55:20,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:20,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:20,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:20,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:20,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:20,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:20,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:55:20,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:20,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:20,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:20,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:20,886 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:20,886 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:20,887 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:20,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:20,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d4c6ec0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:20,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:20,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,890 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:20,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:20,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:20,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:20,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:20,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:20,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:20,892 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:20,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:20,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:20,897 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:20,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:20,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:20,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:20,897 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:20,899 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,899 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:20,899 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:20,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:20,900 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:20,903 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e81c90707da2489cb3dc5a07629ec986 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e81c90707da2489cb3dc5a07629ec986 2023-04-28 08:55:20,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,909 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/ef3abe3229c34cbca1f7e6091873a7ec 2023-04-28 08:55:20,914 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:20,914 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:20,914 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:20,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:20,914 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:20,914 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:20,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:20,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:20,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:20,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:20,915 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:20,915 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:20,915 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:20,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:20,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672180922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:20,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672180923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:20,941 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/f7911ec12a8b473db703722231db8625 2023-04-28 08:55:21,015 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:21,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:21,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:21,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:21,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:21,020 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:21,020 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,020 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:21,020 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:21,020 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,020 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,021 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,021 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,021 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:21,022 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:21,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:21,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:21,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:21,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:21,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,030 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:21,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672181024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,030 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:21,030 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,030 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:21,030 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:21,030 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:21,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:21,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:21,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:21,032 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,033 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,033 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,033 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:21,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e5a469f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:21,037 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:21,037 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,039 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,039 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:21,039 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:21,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:21,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:21,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672181036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,041 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:21,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:21,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:21,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:21,046 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:21,046 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:21,046 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:21,046 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:21,046 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:21,047 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,048 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,048 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:21,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,048 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:21,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:21,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:21,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:21,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:21,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:21,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:21,061 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:21,061 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:21,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:21,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:21,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:21,161 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:21,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:21,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:21,165 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:21,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:21,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:21,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,167 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:21,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:21,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:21,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:21,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:21,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:21,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:21,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:21,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:21,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:21,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:21,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:21,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:21,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:21,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:21,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:21,172 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c4d2f6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:21,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:21,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:21,177 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:21,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:21,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:21,178 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,178 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:21,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:21,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:21,181 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:21,181 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:21,181 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:21,181 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:21,181 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:21,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,183 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:21,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:21,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:21,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:21,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,187 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,187 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:21,187 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:21,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:21,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:21,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:21,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:21,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672181233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672181243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,288 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:21,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:21,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:21,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:21,291 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:21,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:21,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:21,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,293 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:21,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:21,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:21,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:21,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:21,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:21,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:21,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:21,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:21,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:21,304 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,304 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:21,305 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:21,305 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,305 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:21,305 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:21,305 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,305 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:21,305 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:21,308 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,308 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,308 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:21,309 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:21,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@542ae58[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:21,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:21,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:21,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:21,317 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:21,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:21,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:21,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:21,319 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:21,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:21,322 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:21,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:21,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:21,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:21,322 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:21,326 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,326 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,326 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:21,327 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,328 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,335 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,335 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,335 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,335 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,335 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,335 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,335 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,336 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:21,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:21,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:21,336 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:21,336 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:21,336 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:21,336 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:21,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:21,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:21,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/397bbb644c17466b9b4b4dc5dce14661 2023-04-28 08:55:21,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/ef3abe3229c34cbca1f7e6091873a7ec as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ef3abe3229c34cbca1f7e6091873a7ec 2023-04-28 08:55:21,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ef3abe3229c34cbca1f7e6091873a7ec, entries=250, sequenceid=362, filesize=39.2 K 2023-04-28 08:55:21,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/f7911ec12a8b473db703722231db8625 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f7911ec12a8b473db703722231db8625 2023-04-28 08:55:21,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f7911ec12a8b473db703722231db8625, entries=150, sequenceid=362, filesize=11.9 K 2023-04-28 08:55:21,390 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/397bbb644c17466b9b4b4dc5dce14661 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/397bbb644c17466b9b4b4dc5dce14661 2023-04-28 08:55:21,394 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/397bbb644c17466b9b4b4dc5dce14661, entries=150, sequenceid=362, filesize=11.9 K 2023-04-28 08:55:21,395 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5fb523e858d6851c02bb0f82113765f7 in 509ms, sequenceid=362, compaction requested=true 2023-04-28 08:55:21,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:21,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:21,395 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:21,395 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:21,395 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:21,396 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:21,396 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92870 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:21,397 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:21,397 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:21,397 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,397 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:21,397 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e8a3a1901a564a5da1ceb18d6a4c2360, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7b2427f2e33a4f2c8121b8b11cd6e556, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ef3abe3229c34cbca1f7e6091873a7ec] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=90.7 K 2023-04-28 08:55:21,397 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,397 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b99196fcb9804b008d19077527a27c03, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ee8bac3990b84953ade0e59a39465667, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f7911ec12a8b473db703722231db8625] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=36.3 K 2023-04-28 08:55:21,397 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b99196fcb9804b008d19077527a27c03, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682672118322 2023-04-28 08:55:21,397 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e8a3a1901a564a5da1ceb18d6a4c2360, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682672118322 2023-04-28 08:55:21,398 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ee8bac3990b84953ade0e59a39465667, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1682672118666 2023-04-28 08:55:21,398 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7b2427f2e33a4f2c8121b8b11cd6e556, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1682672118666 2023-04-28 08:55:21,399 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting f7911ec12a8b473db703722231db8625, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672120867 2023-04-28 08:55:21,400 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ef3abe3229c34cbca1f7e6091873a7ec, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672120867 2023-04-28 08:55:21,413 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:21,436 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:21,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:21,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:21,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:21,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:21,452 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:21,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:21,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:21,453 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#369 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:21,455 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,455 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:21,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,459 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:21,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:21,461 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,462 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:21,467 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:21,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:21,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:21,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:21,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:21,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:21,467 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:21,467 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,467 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:21,468 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:55:21,468 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:21,468 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:21,468 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:21,468 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:21,468 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:21,469 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:21,503 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304287110bf4322204fbd9c975f86cfe49920 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287110bf4322204fbd9c975f86cfe49920 2023-04-28 08:55:21,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:21,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:21,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672181626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672181627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672181732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672181734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,843 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/fff5d02e27334889955ce15fb81ec435 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fff5d02e27334889955ce15fb81ec435 2023-04-28 08:55:21,847 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into fff5d02e27334889955ce15fb81ec435(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:21,847 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:21,847 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=13, startTime=1682672121395; duration=0sec 2023-04-28 08:55:21,847 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:21,847 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:21,848 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:21,848 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:21,848 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,848 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/995c43be65e543119f8f10f2cc9aa2e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/e245ac5e4c8d44f2bc7016ab1c1951e0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/397bbb644c17466b9b4b4dc5dce14661] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=36.3 K 2023-04-28 08:55:21,849 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 995c43be65e543119f8f10f2cc9aa2e6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1682672118322 2023-04-28 08:55:21,849 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e245ac5e4c8d44f2bc7016ab1c1951e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1682672118666 2023-04-28 08:55:21,849 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 397bbb644c17466b9b4b4dc5dce14661, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672120867 2023-04-28 08:55:21,855 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#371 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:21,917 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/7e8b3661a0bc4e7497aecee844ebae33 2023-04-28 08:55:21,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672181938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:21,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672181939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,946 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/f4e95dd1587045fbb36a30dcf410e54b 2023-04-28 08:55:21,963 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/de0102cb82b54825a3744abebbefa834 2023-04-28 08:55:21,967 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/7e8b3661a0bc4e7497aecee844ebae33 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7e8b3661a0bc4e7497aecee844ebae33 2023-04-28 08:55:21,971 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7e8b3661a0bc4e7497aecee844ebae33, entries=150, sequenceid=374, filesize=25.4 K 2023-04-28 08:55:21,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/f4e95dd1587045fbb36a30dcf410e54b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f4e95dd1587045fbb36a30dcf410e54b 2023-04-28 08:55:21,975 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f4e95dd1587045fbb36a30dcf410e54b, entries=150, sequenceid=374, filesize=11.9 K 2023-04-28 08:55:21,976 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/de0102cb82b54825a3744abebbefa834 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/de0102cb82b54825a3744abebbefa834 2023-04-28 08:55:21,982 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/de0102cb82b54825a3744abebbefa834, entries=150, sequenceid=374, filesize=11.9 K 2023-04-28 08:55:21,983 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5fb523e858d6851c02bb0f82113765f7 in 515ms, sequenceid=374, compaction requested=false 2023-04-28 08:55:21,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:21,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:21,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:21,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:21,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:21,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:21,985 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:21,987 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b76fa5e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:21,987 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:21,987 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:21,987 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,988 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:21,988 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:21,988 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:21,988 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:21,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:21,990 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:21,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:21,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:21,993 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:21,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:21,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:21,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:21,993 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:21,995 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,995 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:21,995 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:21,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:22,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:22,003 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:22,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:22,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:22,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:22,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:22,003 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:22,003 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:22,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:22,103 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:22,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:22,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:22,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:22,106 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:22,106 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:22,107 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:22,107 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:22,107 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:22,107 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:22,108 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:22,108 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:22,109 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,109 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:22,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:22,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:22,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:22,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:22,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:22,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:22,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:22,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:22,110 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:22,110 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:22,110 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:55:22,110 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:22,110 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:22,110 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:22,110 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:22,111 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:22,111 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:22,121 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ccc49e63e0bd45c8a392cb2315aa131b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ccc49e63e0bd45c8a392cb2315aa131b 2023-04-28 08:55:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:22,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:22,265 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/dbfac5442a624c3eb05c766c2f53c2b7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dbfac5442a624c3eb05c766c2f53c2b7 2023-04-28 08:55:22,269 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into dbfac5442a624c3eb05c766c2f53c2b7(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:22,269 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:22,269 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=13, startTime=1682672121396; duration=0sec 2023-04-28 08:55:22,269 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:22,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672182273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672182274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,292 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/f44ad57c1c314b08b78d4e7268055e93 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f44ad57c1c314b08b78d4e7268055e93 2023-04-28 08:55:22,298 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into f44ad57c1c314b08b78d4e7268055e93(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:22,298 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:22,298 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=13, startTime=1682672121395; duration=0sec 2023-04-28 08:55:22,298 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:22,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672182383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672182383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,529 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/19ab4a73cb4f46f8bde79042b7320908 2023-04-28 08:55:22,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672182591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672182591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672182897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:22,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672182898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:22,948 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/cda4d19abd0040aa9e4644ad210e1562 2023-04-28 08:55:23,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:23,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48394 deadline: 1682672183202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,209 DEBUG [Thread-2275] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18234 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:23,358 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/70a8dd9262534f9a8b34ceeb7ced1f6e 2023-04-28 08:55:23,362 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/19ab4a73cb4f46f8bde79042b7320908 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/19ab4a73cb4f46f8bde79042b7320908 2023-04-28 08:55:23,365 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/19ab4a73cb4f46f8bde79042b7320908, entries=150, sequenceid=395, filesize=25.4 K 2023-04-28 08:55:23,366 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/cda4d19abd0040aa9e4644ad210e1562 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/cda4d19abd0040aa9e4644ad210e1562 2023-04-28 08:55:23,370 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/cda4d19abd0040aa9e4644ad210e1562, entries=150, sequenceid=395, filesize=11.9 K 2023-04-28 08:55:23,370 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/70a8dd9262534f9a8b34ceeb7ced1f6e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/70a8dd9262534f9a8b34ceeb7ced1f6e 2023-04-28 08:55:23,374 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/70a8dd9262534f9a8b34ceeb7ced1f6e, entries=150, sequenceid=395, filesize=11.9 K 2023-04-28 08:55:23,374 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 5fb523e858d6851c02bb0f82113765f7 in 1264ms, sequenceid=395, compaction requested=true 2023-04-28 08:55:23,374 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:23,374 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:23,375 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:23,375 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,375 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:23,375 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:23,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:23,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57442b11[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:23,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:23,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,379 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:23,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:23,379 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:23,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:23,380 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:23,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:23,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:23,383 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:23,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:23,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:23,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:23,383 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:23,386 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,386 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,386 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,386 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:23,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:23,390 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,390 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:23,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:23,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:55:23,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:23,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:23,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:23,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:23,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:23,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:23,419 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428bfc0a03cba484492bb7b55db89ba3f93 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bfc0a03cba484492bb7b55db89ba3f93 2023-04-28 08:55:23,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:23,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672183452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:23,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672183453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,490 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:23,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:23,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:23,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:23,493 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:23,493 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,493 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:23,494 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:23,494 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,494 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,495 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,495 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:23,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:23,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:23,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:23,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:23,496 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:23,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:23,497 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,497 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:23,497 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:23,497 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:23,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:23,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:23,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:23,498 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,498 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,499 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,499 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:23,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747106d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:23,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:23,500 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,502 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:23,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:23,502 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:23,502 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,502 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:23,503 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:23,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:23,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:23,505 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:23,505 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:23,505 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:23,505 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:23,505 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:23,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,507 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,508 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,510 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,510 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,510 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,510 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,510 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:23,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:23,510 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:23,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,511 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:23,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,511 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,511 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:23,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:23,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672183558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:23,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672183564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,611 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:23,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:23,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:23,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:23,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:23,616 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:23,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:23,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:23,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,619 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,619 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:23,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:23,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:23,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:23,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:23,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:23,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:23,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:23,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:23,622 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,622 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:23,622 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:23,622 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:23,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:23,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:23,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:23,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,629 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,629 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,629 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:23,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17674eb3[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:23,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:23,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,634 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,634 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:23,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:23,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:23,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:23,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:23,636 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:23,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:23,641 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:23,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:23,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:23,641 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:23,641 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:23,643 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,643 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,643 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,644 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,645 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:23,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:23,648 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,649 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:23,649 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:23,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,649 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,749 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:23,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:23,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:23,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:23,753 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:23,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:23,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:23,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,753 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,754 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:23,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:23,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:23,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:23,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:23,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:23,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:23,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:23,756 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,756 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:23,757 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:23,757 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:23,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:23,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:23,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:23,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,758 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:23,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:23,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b7fe20b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:23,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:23,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:23,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:23,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:23,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:23,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:23,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:23,766 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:23,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:23,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:23,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:23,766 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:23,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:23,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672183763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:23,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:23,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:23,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,773 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:23,773 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:23,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:23,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672183768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,824 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/1e9c58b66a71458cab8e554d6bbaca90 2023-04-28 08:55:23,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/bb321c55f0f54a5788e97bcb771326c5 2023-04-28 08:55:23,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/f6d0aa35d64a447e8c17955e675afd8c 2023-04-28 08:55:23,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/1e9c58b66a71458cab8e554d6bbaca90 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/1e9c58b66a71458cab8e554d6bbaca90 2023-04-28 08:55:23,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/1e9c58b66a71458cab8e554d6bbaca90, entries=200, sequenceid=410, filesize=32.3 K 2023-04-28 08:55:23,873 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:23,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:23,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/bb321c55f0f54a5788e97bcb771326c5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb321c55f0f54a5788e97bcb771326c5 2023-04-28 08:55:23,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:23,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:23,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:23,879 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:23,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:23,879 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:23,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,881 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:23,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb321c55f0f54a5788e97bcb771326c5, entries=150, sequenceid=410, filesize=11.9 K 2023-04-28 08:55:23,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/f6d0aa35d64a447e8c17955e675afd8c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f6d0aa35d64a447e8c17955e675afd8c 2023-04-28 08:55:23,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:23,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:23,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:23,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:23,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:23,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:23,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:23,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:23,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:23,891 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:23,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:23,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:23,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:23,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:23,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d1548d6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:23,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:23,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:23,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:23,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:23,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:23,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,900 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:23,900 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,900 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:23,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:23,904 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:23,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:23,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:23,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:23,904 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:23,905 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,905 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:23,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f6d0aa35d64a447e8c17955e675afd8c, entries=150, sequenceid=410, filesize=11.9 K 2023-04-28 08:55:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,908 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:23,908 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5fb523e858d6851c02bb0f82113765f7 in 500ms, sequenceid=410, compaction requested=true 2023-04-28 08:55:23,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:23,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:23,908 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:23,908 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:23,908 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:23,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,910 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:23,910 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:23,910 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,910 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fff5d02e27334889955ce15fb81ec435, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f4e95dd1587045fbb36a30dcf410e54b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/cda4d19abd0040aa9e4644ad210e1562, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb321c55f0f54a5788e97bcb771326c5] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=48.3 K 2023-04-28 08:55:23,911 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111974 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:23,911 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:23,911 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,911 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting fff5d02e27334889955ce15fb81ec435, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672120867 2023-04-28 08:55:23,911 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f44ad57c1c314b08b78d4e7268055e93, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7e8b3661a0bc4e7497aecee844ebae33, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/19ab4a73cb4f46f8bde79042b7320908, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/1e9c58b66a71458cab8e554d6bbaca90] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=109.3 K 2023-04-28 08:55:23,911 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting f4e95dd1587045fbb36a30dcf410e54b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672120892 2023-04-28 08:55:23,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:23,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:23,912 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:23,912 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f44ad57c1c314b08b78d4e7268055e93, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672120867 2023-04-28 08:55:23,912 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting cda4d19abd0040aa9e4644ad210e1562, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682672121591 2023-04-28 08:55:23,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:23,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:23,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:23,912 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7e8b3661a0bc4e7497aecee844ebae33, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672120892 2023-04-28 08:55:23,912 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:23,912 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:23,913 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bb321c55f0f54a5788e97bcb771326c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1682672122268 2023-04-28 08:55:23,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:23,914 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 19ab4a73cb4f46f8bde79042b7320908, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682672121591 2023-04-28 08:55:23,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:23,916 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1e9c58b66a71458cab8e554d6bbaca90, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1682672122268 2023-04-28 08:55:23,923 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:23,936 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#381 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:23,938 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/820b71f6b6af4fa89cfb67de478de377 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/820b71f6b6af4fa89cfb67de478de377 2023-04-28 08:55:23,942 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into 820b71f6b6af4fa89cfb67de478de377(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:23,942 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:23,942 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=12, startTime=1682672123908; duration=0sec 2023-04-28 08:55:23,942 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:23,942 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:23,943 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:23,943 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:23,943 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:23,943 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dbfac5442a624c3eb05c766c2f53c2b7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/de0102cb82b54825a3744abebbefa834, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/70a8dd9262534f9a8b34ceeb7ced1f6e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f6d0aa35d64a447e8c17955e675afd8c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=48.3 K 2023-04-28 08:55:23,944 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dbfac5442a624c3eb05c766c2f53c2b7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672120867 2023-04-28 08:55:23,944 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting de0102cb82b54825a3744abebbefa834, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1682672120892 2023-04-28 08:55:23,944 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 70a8dd9262534f9a8b34ceeb7ced1f6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682672121591 2023-04-28 08:55:23,945 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting f6d0aa35d64a447e8c17955e675afd8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1682672122268 2023-04-28 08:55:23,950 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#382 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:24,013 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:24,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:24,016 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:24,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:24,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:24,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:24,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,017 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:24,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:24,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:24,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:24,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:24,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:24,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:24,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:24,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:24,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:24,019 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:24,019 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:24,019 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 08:55:24,020 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:24,020 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:24,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:24,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:24,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:24,021 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:24,073 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:24,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672184105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672184115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672184216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672184226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,361 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/da46cfb4da09415fb67fc90970944856 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/da46cfb4da09415fb67fc90970944856 2023-04-28 08:55:24,365 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into da46cfb4da09415fb67fc90970944856(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:24,365 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:24,365 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=12, startTime=1682672123909; duration=0sec 2023-04-28 08:55:24,366 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:24,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672184422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672184433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,442 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042811180616a23b41edb0698c28a2629652 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042811180616a23b41edb0698c28a2629652 2023-04-28 08:55:24,448 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=430, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/80a7e6ab5ef245ab94c4e56419bcd2e6 2023-04-28 08:55:24,481 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/e40c5b7b446c4e99a9a2edc9cadc3600 2023-04-28 08:55:24,508 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/9cee5f7c016046e58d4c9423111031c9 2023-04-28 08:55:24,523 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/80a7e6ab5ef245ab94c4e56419bcd2e6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/80a7e6ab5ef245ab94c4e56419bcd2e6 2023-04-28 08:55:24,527 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/80a7e6ab5ef245ab94c4e56419bcd2e6, entries=150, sequenceid=430, filesize=25.4 K 2023-04-28 08:55:24,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/e40c5b7b446c4e99a9a2edc9cadc3600 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e40c5b7b446c4e99a9a2edc9cadc3600 2023-04-28 08:55:24,534 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e40c5b7b446c4e99a9a2edc9cadc3600, entries=150, sequenceid=430, filesize=11.9 K 2023-04-28 08:55:24,535 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/9cee5f7c016046e58d4c9423111031c9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9cee5f7c016046e58d4c9423111031c9 2023-04-28 08:55:24,540 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9cee5f7c016046e58d4c9423111031c9, entries=150, sequenceid=430, filesize=11.9 K 2023-04-28 08:55:24,540 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5fb523e858d6851c02bb0f82113765f7 in 521ms, sequenceid=430, compaction requested=false 2023-04-28 08:55:24,541 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:24,541 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:24,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:24,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:24,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:24,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:24,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,542 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,543 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,543 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:24,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62eeef61[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:24,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:24,546 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,549 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:24,549 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:24,550 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:24,550 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,550 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:24,551 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:24,551 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:24,551 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:24,551 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:24,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:24,556 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:24,556 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:24,556 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:24,556 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:24,556 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:24,558 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,558 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:24,558 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:24,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,565 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:24,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:24,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:24,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:24,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:24,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:24,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:24,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:24,566 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:24,566 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:24,667 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:24,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:24,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:24,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:24,670 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:24,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:24,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:24,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:24,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:24,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:24,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:24,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:24,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:24,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:24,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:24,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:24,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:24,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:24,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:24,673 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:24,673 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. started... 2023-04-28 08:55:24,673 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:55:24,673 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:24,674 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:24,674 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:24,674 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:24,674 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:24,674 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:24,692 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304282adf248ee8a24d66bff80f443feb2ca9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282adf248ee8a24d66bff80f443feb2ca9 2023-04-28 08:55:24,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:24,732 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. as already flushing 2023-04-28 08:55:24,755 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/f7fdded1e69d40f28c1bd25bb96404c7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f7fdded1e69d40f28c1bd25bb96404c7 2023-04-28 08:55:24,760 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into f7fdded1e69d40f28c1bd25bb96404c7(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:24,760 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:24,760 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=12, startTime=1682672123908; duration=0sec 2023-04-28 08:55:24,760 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:24,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672184792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672184793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672184897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,906 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672184902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,907 DEBUG [Thread-2292] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ce79633 to 127.0.0.1:54655 2023-04-28 08:55:24,907 DEBUG [Thread-2292] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:24,909 DEBUG [Thread-2286] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a35a28d to 127.0.0.1:54655 2023-04-28 08:55:24,909 DEBUG [Thread-2286] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:24,914 DEBUG [Thread-2284] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4366fcdd to 127.0.0.1:54655 2023-04-28 08:55:24,914 DEBUG [Thread-2284] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:24,915 DEBUG [Thread-2290] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0eb8f142 to 127.0.0.1:54655 2023-04-28 08:55:24,915 DEBUG [Thread-2290] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:24,916 DEBUG [Thread-2288] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a1d4246 to 127.0.0.1:54655 2023-04-28 08:55:24,916 DEBUG [Thread-2288] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:24,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48450 deadline: 1682672184938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,940 DEBUG [Thread-2277] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:24,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:48382 deadline: 1682672184954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:24,956 DEBUG [Thread-2273] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18213 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:25,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672185105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:25,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672185107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,110 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=444, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/96c9c827b643436b8fad818444969acd 2023-04-28 08:55:25,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:25,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:25,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672185408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672185408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,519 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/da45119f75e2497287bb3c7c78811aca 2023-04-28 08:55:25,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:25,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:25,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48424 deadline: 1682672185909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:48378 deadline: 1682672185909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,928 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/9e2f7251146b4543b5d9c98690a4bd48 2023-04-28 08:55:25,931 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/96c9c827b643436b8fad818444969acd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/96c9c827b643436b8fad818444969acd 2023-04-28 08:55:25,935 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/96c9c827b643436b8fad818444969acd, entries=150, sequenceid=444, filesize=25.4 K 2023-04-28 08:55:25,936 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/da45119f75e2497287bb3c7c78811aca as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/da45119f75e2497287bb3c7c78811aca 2023-04-28 08:55:25,940 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/da45119f75e2497287bb3c7c78811aca, entries=150, sequenceid=444, filesize=11.9 K 2023-04-28 08:55:25,940 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/9e2f7251146b4543b5d9c98690a4bd48 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9e2f7251146b4543b5d9c98690a4bd48 2023-04-28 08:55:25,944 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9e2f7251146b4543b5d9c98690a4bd48, entries=150, sequenceid=444, filesize=11.9 K 2023-04-28 08:55:25,945 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5fb523e858d6851c02bb0f82113765f7 in 1272ms, sequenceid=444, compaction requested=true 2023-04-28 08:55:25,945 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:25,945 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:25,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:25,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:25,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:25,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:25,947 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:25,948 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,948 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fd13759[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:25,949 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:25,949 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,950 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:25,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:25,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:25,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,950 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:25,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:25,951 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:25,951 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,952 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:25,954 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:25,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:25,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:25,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:25,954 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:25,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:25,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:25,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:25,956 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:25,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:25,956 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:25,957 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:25,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:25,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:25,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:25,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:25,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:25,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:25,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:25,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:25,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:25,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:25,960 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,961 DEBUG [Thread-2283] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:25,961 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:25,961 DEBUG [Thread-2283] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:25,961 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:25,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:26,061 DEBUG [Thread-2283] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:26,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:26,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:26,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:55:26,919 DEBUG [Thread-2279] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06a0df34 to 127.0.0.1:54655 2023-04-28 08:55:26,919 DEBUG [Thread-2279] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:26,920 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:26,920 DEBUG [Thread-2281] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10ee5f06 to 127.0.0.1:54655 2023-04-28 08:55:26,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:26,920 DEBUG [Thread-2281] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:26,920 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:26,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:26,920 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:26,920 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:27,331 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e9d19220d3f241049b2915c603a55a86 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e9d19220d3f241049b2915c603a55a86 2023-04-28 08:55:27,735 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=466, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/e9b6d47beb9d4fcdb144ec2cc344549c 2023-04-28 08:55:28,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/0ac05087785040c3ae977351505bf0ed 2023-04-28 08:55:28,552 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/716c343aec5b4f96b19a464c7b6fc811 2023-04-28 08:55:28,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/e9b6d47beb9d4fcdb144ec2cc344549c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e9b6d47beb9d4fcdb144ec2cc344549c 2023-04-28 08:55:28,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e9b6d47beb9d4fcdb144ec2cc344549c, entries=150, sequenceid=466, filesize=25.4 K 2023-04-28 08:55:28,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/0ac05087785040c3ae977351505bf0ed as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/0ac05087785040c3ae977351505bf0ed 2023-04-28 08:55:28,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/0ac05087785040c3ae977351505bf0ed, entries=150, sequenceid=466, filesize=11.9 K 2023-04-28 08:55:28,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/716c343aec5b4f96b19a464c7b6fc811 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/716c343aec5b4f96b19a464c7b6fc811 2023-04-28 08:55:28,565 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/716c343aec5b4f96b19a464c7b6fc811, entries=150, sequenceid=466, filesize=11.9 K 2023-04-28 08:55:28,566 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=0 B/0 for 5fb523e858d6851c02bb0f82113765f7 in 1647ms, sequenceid=466, compaction requested=true 2023-04-28 08:55:28,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:28,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:28,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:28,566 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:28,566 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:28,566 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:28,567 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105070 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:28,567 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:28,567 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/A is initiating minor compaction (all files) 2023-04-28 08:55:28,567 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/B is initiating minor compaction (all files) 2023-04-28 08:55:28,567 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/A in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:28,567 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/B in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:28,568 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f7fdded1e69d40f28c1bd25bb96404c7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/80a7e6ab5ef245ab94c4e56419bcd2e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/96c9c827b643436b8fad818444969acd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e9b6d47beb9d4fcdb144ec2cc344549c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=102.6 K 2023-04-28 08:55:28,568 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/820b71f6b6af4fa89cfb67de478de377, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e40c5b7b446c4e99a9a2edc9cadc3600, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/da45119f75e2497287bb3c7c78811aca, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/0ac05087785040c3ae977351505bf0ed] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=48.5 K 2023-04-28 08:55:28,568 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f7fdded1e69d40f28c1bd25bb96404c7, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1682672122268 2023-04-28 08:55:28,568 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 820b71f6b6af4fa89cfb67de478de377, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1682672122268 2023-04-28 08:55:28,568 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 80a7e6ab5ef245ab94c4e56419bcd2e6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682672123443 2023-04-28 08:55:28,568 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e40c5b7b446c4e99a9a2edc9cadc3600, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682672123443 2023-04-28 08:55:28,568 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 96c9c827b643436b8fad818444969acd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1682672124096 2023-04-28 08:55:28,568 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting da45119f75e2497287bb3c7c78811aca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1682672124096 2023-04-28 08:55:28,568 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e9b6d47beb9d4fcdb144ec2cc344549c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682672124791 2023-04-28 08:55:28,569 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0ac05087785040c3ae977351505bf0ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682672124791 2023-04-28 08:55:28,575 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#B#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:28,577 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#A#compaction#393 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:28,983 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/044824d284194a7cb9ab12cb1f533d07 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/044824d284194a7cb9ab12cb1f533d07 2023-04-28 08:55:28,987 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/B of 5fb523e858d6851c02bb0f82113765f7 into 044824d284194a7cb9ab12cb1f533d07(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:28,987 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:28,987 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/B, priority=12, startTime=1682672128566; duration=0sec 2023-04-28 08:55:28,987 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:28,988 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:28,988 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:28,988 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 5fb523e858d6851c02bb0f82113765f7/C is initiating minor compaction (all files) 2023-04-28 08:55:28,988 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5fb523e858d6851c02bb0f82113765f7/C in TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:28,989 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/da46cfb4da09415fb67fc90970944856, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9cee5f7c016046e58d4c9423111031c9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9e2f7251146b4543b5d9c98690a4bd48, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/716c343aec5b4f96b19a464c7b6fc811] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp, totalSize=48.5 K 2023-04-28 08:55:28,989 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting da46cfb4da09415fb67fc90970944856, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1682672122268 2023-04-28 08:55:28,989 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9cee5f7c016046e58d4c9423111031c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1682672123443 2023-04-28 08:55:28,989 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9e2f7251146b4543b5d9c98690a4bd48, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1682672124096 2023-04-28 08:55:28,989 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 716c343aec5b4f96b19a464c7b6fc811, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1682672124791 2023-04-28 08:55:28,995 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 5fb523e858d6851c02bb0f82113765f7#C#compaction#394 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:29,406 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/9d9d72e4be3b4247847fa10a57e68376 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9d9d72e4be3b4247847fa10a57e68376 2023-04-28 08:55:29,410 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/C of 5fb523e858d6851c02bb0f82113765f7 into 9d9d72e4be3b4247847fa10a57e68376(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:29,410 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:29,410 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/C, priority=12, startTime=1682672128566; duration=0sec 2023-04-28 08:55:29,410 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:29,793 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/6d36c53ae4bd42f9a77e40781a0a6f12 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/6d36c53ae4bd42f9a77e40781a0a6f12 2023-04-28 08:55:29,797 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5fb523e858d6851c02bb0f82113765f7/A of 5fb523e858d6851c02bb0f82113765f7 into 6d36c53ae4bd42f9a77e40781a0a6f12(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-28 08:55:29,797 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:29,797 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7., storeName=5fb523e858d6851c02bb0f82113765f7/A, priority=12, startTime=1682672128566; duration=1sec 2023-04-28 08:55:29,797 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:33,277 DEBUG [Thread-2275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x419b69bf to 127.0.0.1:54655 2023-04-28 08:55:33,277 DEBUG [Thread-2275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:34,994 DEBUG [Thread-2273] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c8d927d to 127.0.0.1:54655 2023-04-28 08:55:34,994 DEBUG [Thread-2273] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:35,030 DEBUG [Thread-2277] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c6cac96 to 127.0.0.1:54655 2023-04-28 08:55:35,030 DEBUG [Thread-2277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:35,030 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 162 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 178 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1991 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5970 rows 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1967 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5900 rows 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1961 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5883 rows 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1978 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5934 rows 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1982 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 5946 rows 2023-04-28 08:55:35,031 INFO [Listener at localhost/39717] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 08:55:35,031 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49b5bd0d to 127.0.0.1:54655 2023-04-28 08:55:35,031 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:35,035 INFO [Listener at localhost/39717] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 08:55:35,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable TestAcidGuarantees 2023-04-28 08:55:35,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:35,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 08:55:35,039 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672135039"}]},"ts":"1682672135039"} 2023-04-28 08:55:35,040 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 08:55:35,042 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 08:55:35,042 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, UNASSIGN}] 2023-04-28 08:55:35,043 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, UNASSIGN 2023-04-28 08:55:35,044 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:35,044 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672135044"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672135044"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672135044"}]},"ts":"1682672135044"} 2023-04-28 08:55:35,045 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:55:35,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 08:55:35,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:35,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 5fb523e858d6851c02bb0f82113765f7, disabling compactions & flushes 2023-04-28 08:55:35,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:35,196 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. after waiting 0 ms 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:35,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 5fb523e858d6851c02bb0f82113765f7 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=A 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=B 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5fb523e858d6851c02bb0f82113765f7, store=C 2023-04-28 08:55:35,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:35,206 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428a33248571e2e452e8a285e146f706a64 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a33248571e2e452e8a285e146f706a64 2023-04-28 08:55:35,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 08:55:35,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=475, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/3bd3a555265a48b1ae5348b06bd9e25f 2023-04-28 08:55:35,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 08:55:36,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/68a607c98814410383e46a3d4faa4d44 2023-04-28 08:55:36,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 08:55:36,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/f965cdda17554d518f61f69bdfed8881 2023-04-28 08:55:36,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/A/3bd3a555265a48b1ae5348b06bd9e25f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3bd3a555265a48b1ae5348b06bd9e25f 2023-04-28 08:55:36,435 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3bd3a555265a48b1ae5348b06bd9e25f, entries=100, sequenceid=475, filesize=18.6 K 2023-04-28 08:55:36,435 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/B/68a607c98814410383e46a3d4faa4d44 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/68a607c98814410383e46a3d4faa4d44 2023-04-28 08:55:36,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/68a607c98814410383e46a3d4faa4d44, entries=100, sequenceid=475, filesize=9.5 K 2023-04-28 08:55:36,439 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/.tmp/C/f965cdda17554d518f61f69bdfed8881 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f965cdda17554d518f61f69bdfed8881 2023-04-28 08:55:36,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f965cdda17554d518f61f69bdfed8881, entries=100, sequenceid=475, filesize=9.5 K 2023-04-28 08:55:36,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 5fb523e858d6851c02bb0f82113765f7 in 1246ms, sequenceid=475, compaction requested=false 2023-04-28 08:55:36,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/940844700ec1435d8913551657b6861e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/44bd1633f80646f58da0a344bf47ea81, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/836e322f0ba8456cb6cb65b65999f889, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/590a6b534d514eeeb0f1e685f2456d70, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3520118f451949199b8a8f0ceb88f4aa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ae0050181c6a489fb636b3dd33a8602f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e117af3e49204d8f9ab679e6dfb50509, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c8df26c2f8424f5bb37136c6e3125755, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/5ae8239fa4704646a636a46f2956a319, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/06cae32d8f0c4b8a9bc86fad597aadd3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3db7290a37ff46b1b19c9c7827095708, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/9c5c95ff5b09406f97350b2ff3a0fd13, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/473d4b7e01cd42ba8e69a8b505ed22b5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/cb49d5944ba34b1e943e5932510e8498, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/83d8ceabf37149a4be2ae50f89f9b8dd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7a8c954848b84a19a8f3e56ea88bf222, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/d5e07cca93914a3f98390a90a7716082, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c606c00f288f4c95b0867131f150c403, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/60ff54f91cb24d599418b7d9d9e07077, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e8a3a1901a564a5da1ceb18d6a4c2360, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7b2427f2e33a4f2c8121b8b11cd6e556, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ef3abe3229c34cbca1f7e6091873a7ec, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f44ad57c1c314b08b78d4e7268055e93, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7e8b3661a0bc4e7497aecee844ebae33, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/19ab4a73cb4f46f8bde79042b7320908, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/1e9c58b66a71458cab8e554d6bbaca90, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f7fdded1e69d40f28c1bd25bb96404c7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/80a7e6ab5ef245ab94c4e56419bcd2e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/96c9c827b643436b8fad818444969acd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e9b6d47beb9d4fcdb144ec2cc344549c] to archive 2023-04-28 08:55:36,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:55:36,466 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/940844700ec1435d8913551657b6861e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/940844700ec1435d8913551657b6861e 2023-04-28 08:55:36,467 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/44bd1633f80646f58da0a344bf47ea81 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/44bd1633f80646f58da0a344bf47ea81 2023-04-28 08:55:36,468 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/836e322f0ba8456cb6cb65b65999f889 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/836e322f0ba8456cb6cb65b65999f889 2023-04-28 08:55:36,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/590a6b534d514eeeb0f1e685f2456d70 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/590a6b534d514eeeb0f1e685f2456d70 2023-04-28 08:55:36,470 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3520118f451949199b8a8f0ceb88f4aa to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3520118f451949199b8a8f0ceb88f4aa 2023-04-28 08:55:36,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ae0050181c6a489fb636b3dd33a8602f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ae0050181c6a489fb636b3dd33a8602f 2023-04-28 08:55:36,472 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e117af3e49204d8f9ab679e6dfb50509 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e117af3e49204d8f9ab679e6dfb50509 2023-04-28 08:55:36,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c8df26c2f8424f5bb37136c6e3125755 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c8df26c2f8424f5bb37136c6e3125755 2023-04-28 08:55:36,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/5ae8239fa4704646a636a46f2956a319 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/5ae8239fa4704646a636a46f2956a319 2023-04-28 08:55:36,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/06cae32d8f0c4b8a9bc86fad597aadd3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/06cae32d8f0c4b8a9bc86fad597aadd3 2023-04-28 08:55:36,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3db7290a37ff46b1b19c9c7827095708 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3db7290a37ff46b1b19c9c7827095708 2023-04-28 08:55:36,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/9c5c95ff5b09406f97350b2ff3a0fd13 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/9c5c95ff5b09406f97350b2ff3a0fd13 2023-04-28 08:55:36,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/473d4b7e01cd42ba8e69a8b505ed22b5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/473d4b7e01cd42ba8e69a8b505ed22b5 2023-04-28 08:55:36,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/cb49d5944ba34b1e943e5932510e8498 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/cb49d5944ba34b1e943e5932510e8498 2023-04-28 08:55:36,479 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/83d8ceabf37149a4be2ae50f89f9b8dd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/83d8ceabf37149a4be2ae50f89f9b8dd 2023-04-28 08:55:36,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7a8c954848b84a19a8f3e56ea88bf222 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7a8c954848b84a19a8f3e56ea88bf222 2023-04-28 08:55:36,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/d5e07cca93914a3f98390a90a7716082 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/d5e07cca93914a3f98390a90a7716082 2023-04-28 08:55:36,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c606c00f288f4c95b0867131f150c403 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/c606c00f288f4c95b0867131f150c403 2023-04-28 08:55:36,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/60ff54f91cb24d599418b7d9d9e07077 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/60ff54f91cb24d599418b7d9d9e07077 2023-04-28 08:55:36,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e8a3a1901a564a5da1ceb18d6a4c2360 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e8a3a1901a564a5da1ceb18d6a4c2360 2023-04-28 08:55:36,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7b2427f2e33a4f2c8121b8b11cd6e556 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7b2427f2e33a4f2c8121b8b11cd6e556 2023-04-28 08:55:36,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ef3abe3229c34cbca1f7e6091873a7ec to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/ef3abe3229c34cbca1f7e6091873a7ec 2023-04-28 08:55:36,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f44ad57c1c314b08b78d4e7268055e93 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f44ad57c1c314b08b78d4e7268055e93 2023-04-28 08:55:36,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7e8b3661a0bc4e7497aecee844ebae33 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/7e8b3661a0bc4e7497aecee844ebae33 2023-04-28 08:55:36,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/19ab4a73cb4f46f8bde79042b7320908 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/19ab4a73cb4f46f8bde79042b7320908 2023-04-28 08:55:36,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/1e9c58b66a71458cab8e554d6bbaca90 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/1e9c58b66a71458cab8e554d6bbaca90 2023-04-28 08:55:36,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f7fdded1e69d40f28c1bd25bb96404c7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/f7fdded1e69d40f28c1bd25bb96404c7 2023-04-28 08:55:36,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/80a7e6ab5ef245ab94c4e56419bcd2e6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/80a7e6ab5ef245ab94c4e56419bcd2e6 2023-04-28 08:55:36,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/96c9c827b643436b8fad818444969acd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/96c9c827b643436b8fad818444969acd 2023-04-28 08:55:36,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e9b6d47beb9d4fcdb144ec2cc344549c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/e9b6d47beb9d4fcdb144ec2cc344549c 2023-04-28 08:55:36,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/025bf7226e1843a18b0feecf885b86ba, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fa3176ddbf244df095aa5ce59e28e4f7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e65cdb8513fd492faf498fc2c2df1c8a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49dfbcde51a44d780947cf585aa6155, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ded4cedac0274a22974108aca2ebc0b5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49ec9d1b65e4efe9bb661604829165c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/90b722e5cc4a41d886c6ad080b4ffd41, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dc9a97d104a94afb9eb969a376f3ffcb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/a83662db67fc43e0a2d189e0f1a1d82b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fc0bff06f9ba4f4b86b46858a182dbe2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b654271ff12b41839ebab21a2a8b601c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb4927774a624cd2bf38ec1b1c262260, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dde6d6d534a640f9a81edce8704eabe7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/719d5af07238429eae98a06c7bf444df, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/9476861abecb4106bc08acfbcbf5099c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/d3d2f6873dfa4c25a318533b546bce4a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/56d52410ba1e44eda6283d55c04be309, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/6a3b7b808ddd44bd98e593ef77fe56b5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b99196fcb9804b008d19077527a27c03, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ddaeadf638684f438510e99dea493b2e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ee8bac3990b84953ade0e59a39465667, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fff5d02e27334889955ce15fb81ec435, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f7911ec12a8b473db703722231db8625, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f4e95dd1587045fbb36a30dcf410e54b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/cda4d19abd0040aa9e4644ad210e1562, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/820b71f6b6af4fa89cfb67de478de377, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb321c55f0f54a5788e97bcb771326c5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e40c5b7b446c4e99a9a2edc9cadc3600, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/da45119f75e2497287bb3c7c78811aca, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/0ac05087785040c3ae977351505bf0ed] to archive 2023-04-28 08:55:36,496 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:55:36,497 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/025bf7226e1843a18b0feecf885b86ba to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/025bf7226e1843a18b0feecf885b86ba 2023-04-28 08:55:36,498 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fa3176ddbf244df095aa5ce59e28e4f7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fa3176ddbf244df095aa5ce59e28e4f7 2023-04-28 08:55:36,498 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e65cdb8513fd492faf498fc2c2df1c8a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e65cdb8513fd492faf498fc2c2df1c8a 2023-04-28 08:55:36,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49dfbcde51a44d780947cf585aa6155 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49dfbcde51a44d780947cf585aa6155 2023-04-28 08:55:36,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ded4cedac0274a22974108aca2ebc0b5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ded4cedac0274a22974108aca2ebc0b5 2023-04-28 08:55:36,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49ec9d1b65e4efe9bb661604829165c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/c49ec9d1b65e4efe9bb661604829165c 2023-04-28 08:55:36,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/90b722e5cc4a41d886c6ad080b4ffd41 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/90b722e5cc4a41d886c6ad080b4ffd41 2023-04-28 08:55:36,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dc9a97d104a94afb9eb969a376f3ffcb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dc9a97d104a94afb9eb969a376f3ffcb 2023-04-28 08:55:36,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/a83662db67fc43e0a2d189e0f1a1d82b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/a83662db67fc43e0a2d189e0f1a1d82b 2023-04-28 08:55:36,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fc0bff06f9ba4f4b86b46858a182dbe2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fc0bff06f9ba4f4b86b46858a182dbe2 2023-04-28 08:55:36,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b654271ff12b41839ebab21a2a8b601c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b654271ff12b41839ebab21a2a8b601c 2023-04-28 08:55:36,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb4927774a624cd2bf38ec1b1c262260 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb4927774a624cd2bf38ec1b1c262260 2023-04-28 08:55:36,507 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dde6d6d534a640f9a81edce8704eabe7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/dde6d6d534a640f9a81edce8704eabe7 2023-04-28 08:55:36,508 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/719d5af07238429eae98a06c7bf444df to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/719d5af07238429eae98a06c7bf444df 2023-04-28 08:55:36,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/9476861abecb4106bc08acfbcbf5099c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/9476861abecb4106bc08acfbcbf5099c 2023-04-28 08:55:36,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/d3d2f6873dfa4c25a318533b546bce4a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/d3d2f6873dfa4c25a318533b546bce4a 2023-04-28 08:55:36,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/56d52410ba1e44eda6283d55c04be309 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/56d52410ba1e44eda6283d55c04be309 2023-04-28 08:55:36,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/6a3b7b808ddd44bd98e593ef77fe56b5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/6a3b7b808ddd44bd98e593ef77fe56b5 2023-04-28 08:55:36,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b99196fcb9804b008d19077527a27c03 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/b99196fcb9804b008d19077527a27c03 2023-04-28 08:55:36,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ddaeadf638684f438510e99dea493b2e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ddaeadf638684f438510e99dea493b2e 2023-04-28 08:55:36,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ee8bac3990b84953ade0e59a39465667 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/ee8bac3990b84953ade0e59a39465667 2023-04-28 08:55:36,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fff5d02e27334889955ce15fb81ec435 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/fff5d02e27334889955ce15fb81ec435 2023-04-28 08:55:36,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f7911ec12a8b473db703722231db8625 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f7911ec12a8b473db703722231db8625 2023-04-28 08:55:36,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f4e95dd1587045fbb36a30dcf410e54b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/f4e95dd1587045fbb36a30dcf410e54b 2023-04-28 08:55:36,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/cda4d19abd0040aa9e4644ad210e1562 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/cda4d19abd0040aa9e4644ad210e1562 2023-04-28 08:55:36,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/820b71f6b6af4fa89cfb67de478de377 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/820b71f6b6af4fa89cfb67de478de377 2023-04-28 08:55:36,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb321c55f0f54a5788e97bcb771326c5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/bb321c55f0f54a5788e97bcb771326c5 2023-04-28 08:55:36,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e40c5b7b446c4e99a9a2edc9cadc3600 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/e40c5b7b446c4e99a9a2edc9cadc3600 2023-04-28 08:55:36,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/da45119f75e2497287bb3c7c78811aca to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/da45119f75e2497287bb3c7c78811aca 2023-04-28 08:55:36,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/0ac05087785040c3ae977351505bf0ed to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/0ac05087785040c3ae977351505bf0ed 2023-04-28 08:55:36,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/fbd3c6c5cba3460a810a1dd22013d45f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/8fa64e7eec5b4648aecfa3d843fb0053, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7a81843602924aef97fac0dcaed71953, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/aa2db71d3fc84126acc158fbca44b861, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/03cc31afcda64c0397b7cf6a5a9b8c12, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3c2ed61ef2ef448e800e744164f29589, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b24b402e1bcc462d95dd25f06c021cb6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dde1bcf9c66d4c19aef03db62013851d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/daca2ec0428448ee81c6c95d4b473b24, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/97cc52fe20dc4787bf3a5dede4080b1c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7ed5b2fcda9b4da0b91de9ba659eff1e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/94fc842d8c18429fa5ffbbf4be04cf86, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/4be180ed2cce4a6bb07a1a9e89b6e1de, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/173d20a077654d859e104948f683a3a1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b1c67bde72d84612a9d9424ce2cd61d3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/ffb6826b5daa4135a981d06bd9d4a92f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/96213c5595b94cf6aa81c32f8e547f8a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3d028c5059c84f7e96e68fdc723a4acf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/995c43be65e543119f8f10f2cc9aa2e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/d057df3d14664778a4a821e4f13aba7a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/e245ac5e4c8d44f2bc7016ab1c1951e0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dbfac5442a624c3eb05c766c2f53c2b7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/397bbb644c17466b9b4b4dc5dce14661, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/de0102cb82b54825a3744abebbefa834, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/70a8dd9262534f9a8b34ceeb7ced1f6e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/da46cfb4da09415fb67fc90970944856, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f6d0aa35d64a447e8c17955e675afd8c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9cee5f7c016046e58d4c9423111031c9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9e2f7251146b4543b5d9c98690a4bd48, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/716c343aec5b4f96b19a464c7b6fc811] to archive 2023-04-28 08:55:36,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:55:36,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/fbd3c6c5cba3460a810a1dd22013d45f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/fbd3c6c5cba3460a810a1dd22013d45f 2023-04-28 08:55:36,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/8fa64e7eec5b4648aecfa3d843fb0053 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/8fa64e7eec5b4648aecfa3d843fb0053 2023-04-28 08:55:36,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7a81843602924aef97fac0dcaed71953 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7a81843602924aef97fac0dcaed71953 2023-04-28 08:55:36,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/aa2db71d3fc84126acc158fbca44b861 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/aa2db71d3fc84126acc158fbca44b861 2023-04-28 08:55:36,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/03cc31afcda64c0397b7cf6a5a9b8c12 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/03cc31afcda64c0397b7cf6a5a9b8c12 2023-04-28 08:55:36,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3c2ed61ef2ef448e800e744164f29589 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3c2ed61ef2ef448e800e744164f29589 2023-04-28 08:55:36,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b24b402e1bcc462d95dd25f06c021cb6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b24b402e1bcc462d95dd25f06c021cb6 2023-04-28 08:55:36,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dde1bcf9c66d4c19aef03db62013851d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dde1bcf9c66d4c19aef03db62013851d 2023-04-28 08:55:36,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/daca2ec0428448ee81c6c95d4b473b24 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/daca2ec0428448ee81c6c95d4b473b24 2023-04-28 08:55:36,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/97cc52fe20dc4787bf3a5dede4080b1c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/97cc52fe20dc4787bf3a5dede4080b1c 2023-04-28 08:55:36,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7ed5b2fcda9b4da0b91de9ba659eff1e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/7ed5b2fcda9b4da0b91de9ba659eff1e 2023-04-28 08:55:36,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/94fc842d8c18429fa5ffbbf4be04cf86 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/94fc842d8c18429fa5ffbbf4be04cf86 2023-04-28 08:55:36,546 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/4be180ed2cce4a6bb07a1a9e89b6e1de to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/4be180ed2cce4a6bb07a1a9e89b6e1de 2023-04-28 08:55:36,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/173d20a077654d859e104948f683a3a1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/173d20a077654d859e104948f683a3a1 2023-04-28 08:55:36,548 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b1c67bde72d84612a9d9424ce2cd61d3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/b1c67bde72d84612a9d9424ce2cd61d3 2023-04-28 08:55:36,549 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/ffb6826b5daa4135a981d06bd9d4a92f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/ffb6826b5daa4135a981d06bd9d4a92f 2023-04-28 08:55:36,550 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/96213c5595b94cf6aa81c32f8e547f8a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/96213c5595b94cf6aa81c32f8e547f8a 2023-04-28 08:55:36,551 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3d028c5059c84f7e96e68fdc723a4acf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/3d028c5059c84f7e96e68fdc723a4acf 2023-04-28 08:55:36,552 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/995c43be65e543119f8f10f2cc9aa2e6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/995c43be65e543119f8f10f2cc9aa2e6 2023-04-28 08:55:36,553 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/d057df3d14664778a4a821e4f13aba7a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/d057df3d14664778a4a821e4f13aba7a 2023-04-28 08:55:36,554 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/e245ac5e4c8d44f2bc7016ab1c1951e0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/e245ac5e4c8d44f2bc7016ab1c1951e0 2023-04-28 08:55:36,555 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dbfac5442a624c3eb05c766c2f53c2b7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/dbfac5442a624c3eb05c766c2f53c2b7 2023-04-28 08:55:36,556 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/397bbb644c17466b9b4b4dc5dce14661 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/397bbb644c17466b9b4b4dc5dce14661 2023-04-28 08:55:36,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/de0102cb82b54825a3744abebbefa834 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/de0102cb82b54825a3744abebbefa834 2023-04-28 08:55:36,558 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/70a8dd9262534f9a8b34ceeb7ced1f6e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/70a8dd9262534f9a8b34ceeb7ced1f6e 2023-04-28 08:55:36,559 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/da46cfb4da09415fb67fc90970944856 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/da46cfb4da09415fb67fc90970944856 2023-04-28 08:55:36,561 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f6d0aa35d64a447e8c17955e675afd8c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f6d0aa35d64a447e8c17955e675afd8c 2023-04-28 08:55:36,562 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9cee5f7c016046e58d4c9423111031c9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9cee5f7c016046e58d4c9423111031c9 2023-04-28 08:55:36,563 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9e2f7251146b4543b5d9c98690a4bd48 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9e2f7251146b4543b5d9c98690a4bd48 2023-04-28 08:55:36,565 DEBUG [StoreCloser-TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/716c343aec5b4f96b19a464c7b6fc811 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/716c343aec5b4f96b19a464c7b6fc811 2023-04-28 08:55:36,570 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/recovered.edits/478.seqid, newMaxSeqId=478, maxSeqId=4 2023-04-28 08:55:36,572 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7. 2023-04-28 08:55:36,572 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 5fb523e858d6851c02bb0f82113765f7: 2023-04-28 08:55:36,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:36,574 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=5fb523e858d6851c02bb0f82113765f7, regionState=CLOSED 2023-04-28 08:55:36,574 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672136574"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672136574"}]},"ts":"1682672136574"} 2023-04-28 08:55:36,576 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-28 08:55:36,576 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 5fb523e858d6851c02bb0f82113765f7, server=jenkins-hbase6.apache.org,40909,1682672018517 in 1.5300 sec 2023-04-28 08:55:36,577 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-28 08:55:36,577 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5fb523e858d6851c02bb0f82113765f7, UNASSIGN in 1.5340 sec 2023-04-28 08:55:36,578 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672136578"}]},"ts":"1682672136578"} 2023-04-28 08:55:36,579 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 08:55:36,580 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 08:55:36,581 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5450 sec 2023-04-28 08:55:37,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-28 08:55:37,143 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-28 08:55:37,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete TestAcidGuarantees 2023-04-28 08:55:37,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:37,147 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:37,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-28 08:55:37,147 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:37,152 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:37,154 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/recovered.edits] 2023-04-28 08:55:37,157 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3bd3a555265a48b1ae5348b06bd9e25f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/3bd3a555265a48b1ae5348b06bd9e25f 2023-04-28 08:55:37,158 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/6d36c53ae4bd42f9a77e40781a0a6f12 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/A/6d36c53ae4bd42f9a77e40781a0a6f12 2023-04-28 08:55:37,161 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/044824d284194a7cb9ab12cb1f533d07 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/044824d284194a7cb9ab12cb1f533d07 2023-04-28 08:55:37,162 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/68a607c98814410383e46a3d4faa4d44 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/B/68a607c98814410383e46a3d4faa4d44 2023-04-28 08:55:37,164 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9d9d72e4be3b4247847fa10a57e68376 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/9d9d72e4be3b4247847fa10a57e68376 2023-04-28 08:55:37,165 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f965cdda17554d518f61f69bdfed8881 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/C/f965cdda17554d518f61f69bdfed8881 2023-04-28 08:55:37,168 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/recovered.edits/478.seqid to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7/recovered.edits/478.seqid 2023-04-28 08:55:37,169 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/5fb523e858d6851c02bb0f82113765f7 2023-04-28 08:55:37,169 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:55:37,169 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 08:55:37,170 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 08:55:37,173 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280107ff63578d497881ee0799ef06ba02 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280107ff63578d497881ee0799ef06ba02 2023-04-28 08:55:37,175 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042804cf01986495408ebc9e278a6dcce483 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042804cf01986495408ebc9e278a6dcce483 2023-04-28 08:55:37,176 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042811180616a23b41edb0698c28a2629652 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042811180616a23b41edb0698c28a2629652 2023-04-28 08:55:37,177 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282adf248ee8a24d66bff80f443feb2ca9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282adf248ee8a24d66bff80f443feb2ca9 2023-04-28 08:55:37,179 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042851d0615e407747d6be9e04551f961600 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042851d0615e407747d6be9e04551f961600 2023-04-28 08:55:37,180 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428527beb3916094560b02ba29377f9ada1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428527beb3916094560b02ba29377f9ada1 2023-04-28 08:55:37,181 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428626a6391a01e4e79b18730dd5dced8ac to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428626a6391a01e4e79b18730dd5dced8ac 2023-04-28 08:55:37,182 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287110bf4322204fbd9c975f86cfe49920 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304287110bf4322204fbd9c975f86cfe49920 2023-04-28 08:55:37,183 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042872bdafd9ce254b8c833e44f1673d9e10 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042872bdafd9ce254b8c833e44f1673d9e10 2023-04-28 08:55:37,184 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042874055bcb515a41c2ac55b7a4a93a6aa7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042874055bcb515a41c2ac55b7a4a93a6aa7 2023-04-28 08:55:37,186 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289843ef6c2a7945fa978e09200258c8ab to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304289843ef6c2a7945fa978e09200258c8ab 2023-04-28 08:55:37,187 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a05e6b3fdddb4eb686c4606dc5f09338 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a05e6b3fdddb4eb686c4606dc5f09338 2023-04-28 08:55:37,188 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a33248571e2e452e8a285e146f706a64 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428a33248571e2e452e8a285e146f706a64 2023-04-28 08:55:37,189 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ac90ee39eda24e9ca166b0f8a893b76e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ac90ee39eda24e9ca166b0f8a893b76e 2023-04-28 08:55:37,190 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aedf8f2b5fc04d0293200a21fdd2cc16 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428aedf8f2b5fc04d0293200a21fdd2cc16 2023-04-28 08:55:37,191 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5c94a3566cb40139f53aef90766e9a5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5c94a3566cb40139f53aef90766e9a5 2023-04-28 08:55:37,192 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bfc0a03cba484492bb7b55db89ba3f93 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428bfc0a03cba484492bb7b55db89ba3f93 2023-04-28 08:55:37,193 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ccc49e63e0bd45c8a392cb2315aa131b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ccc49e63e0bd45c8a392cb2315aa131b 2023-04-28 08:55:37,194 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d2c827b18b3641f59dff38cee3fa6050 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d2c827b18b3641f59dff38cee3fa6050 2023-04-28 08:55:37,195 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d97008e8a32f4337954a7b44e4a6a2af to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d97008e8a32f4337954a7b44e4a6a2af 2023-04-28 08:55:37,196 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e81c90707da2489cb3dc5a07629ec986 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e81c90707da2489cb3dc5a07629ec986 2023-04-28 08:55:37,197 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e95fefb7cb4d401b8ead9ce30398b0f9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e95fefb7cb4d401b8ead9ce30398b0f9 2023-04-28 08:55:37,198 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e9d19220d3f241049b2915c603a55a86 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e9d19220d3f241049b2915c603a55a86 2023-04-28 08:55:37,200 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f7f213d292b44cf6b90ddc1dc403b90d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f7f213d292b44cf6b90ddc1dc403b90d 2023-04-28 08:55:37,200 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 08:55:37,202 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:37,204 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 08:55:37,205 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 08:55:37,206 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:37,206 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 08:55:37,206 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682672137206"}]},"ts":"9223372036854775807"} 2023-04-28 08:55:37,212 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 08:55:37,212 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5fb523e858d6851c02bb0f82113765f7, NAME => 'TestAcidGuarantees,,1682672101786.5fb523e858d6851c02bb0f82113765f7.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 08:55:37,212 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 08:55:37,212 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682672137212"}]},"ts":"9223372036854775807"} 2023-04-28 08:55:37,213 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 08:55:37,215 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:37,216 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 71 msec 2023-04-28 08:55:37,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-28 08:55:37,248 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-28 08:55:37,261 INFO [Listener at localhost/39717] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224 (was 228), OpenFileDescriptor=518 (was 556), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 423), ProcessCount=170 (was 170), AvailableMemoryMB=6027 (was 6080) 2023-04-28 08:55:37,274 INFO [Listener at localhost/39717] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=224, OpenFileDescriptor=518, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=170, AvailableMemoryMB=6026 2023-04-28 08:55:37,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:55:37,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:55:37,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 08:55:37,279 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 08:55:37,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-28 08:55:37,280 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 08:55:37,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 08:55:37,283 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:37,283 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4 empty. 2023-04-28 08:55:37,284 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:37,284 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:55:37,293 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:55:37,295 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2ff295b65a6980314fe434234aeef4a4, NAME => 'TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp 2023-04-28 08:55:37,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 08:55:37,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 08:55:37,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:55:37,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2ff295b65a6980314fe434234aeef4a4, disabling compactions & flushes 2023-04-28 08:55:37,710 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:37,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:37,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. after waiting 0 ms 2023-04-28 08:55:37,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:37,710 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:37,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:37,712 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 08:55:37,713 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672137713"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672137713"}]},"ts":"1682672137713"} 2023-04-28 08:55:37,714 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 08:55:37,714 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 08:55:37,715 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672137715"}]},"ts":"1682672137715"} 2023-04-28 08:55:37,715 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 08:55:37,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ff295b65a6980314fe434234aeef4a4, ASSIGN}] 2023-04-28 08:55:37,720 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ff295b65a6980314fe434234aeef4a4, ASSIGN 2023-04-28 08:55:37,721 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ff295b65a6980314fe434234aeef4a4, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=false 2023-04-28 08:55:37,872 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2ff295b65a6980314fe434234aeef4a4, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:37,872 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672137872"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672137872"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672137872"}]},"ts":"1682672137872"} 2023-04-28 08:55:37,873 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:55:37,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 08:55:38,028 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:38,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2ff295b65a6980314fe434234aeef4a4, NAME => 'TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:55:38,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:55:38,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,030 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,031 DEBUG [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A 2023-04-28 08:55:38,031 DEBUG [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A 2023-04-28 08:55:38,031 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:38,031 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ff295b65a6980314fe434234aeef4a4 columnFamilyName A 2023-04-28 08:55:38,032 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.HStore(310): Store=2ff295b65a6980314fe434234aeef4a4/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:38,032 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,033 DEBUG [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B 2023-04-28 08:55:38,033 DEBUG [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B 2023-04-28 08:55:38,033 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:38,033 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ff295b65a6980314fe434234aeef4a4 columnFamilyName B 2023-04-28 08:55:38,034 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.HStore(310): Store=2ff295b65a6980314fe434234aeef4a4/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:38,034 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,035 DEBUG [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C 2023-04-28 08:55:38,035 DEBUG [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C 2023-04-28 08:55:38,035 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:55:38,035 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ff295b65a6980314fe434234aeef4a4 columnFamilyName C 2023-04-28 08:55:38,035 INFO [StoreOpener-2ff295b65a6980314fe434234aeef4a4-1] regionserver.HStore(310): Store=2ff295b65a6980314fe434234aeef4a4/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:55:38,035 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:38,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,036 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:55:38,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,040 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:55:38,041 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 2ff295b65a6980314fe434234aeef4a4; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61483524, jitterRate=-0.08382409811019897}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:55:38,041 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:38,041 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., pid=49, masterSystemTime=1682672138025 2023-04-28 08:55:38,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:38,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:38,043 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2ff295b65a6980314fe434234aeef4a4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,043 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672138043"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672138043"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672138043"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672138043"}]},"ts":"1682672138043"} 2023-04-28 08:55:38,045 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-28 08:55:38,045 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 in 171 msec 2023-04-28 08:55:38,046 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-28 08:55:38,046 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ff295b65a6980314fe434234aeef4a4, ASSIGN in 326 msec 2023-04-28 08:55:38,047 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 08:55:38,047 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672138047"}]},"ts":"1682672138047"} 2023-04-28 08:55:38,048 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 08:55:38,049 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 08:55:38,050 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 772 msec 2023-04-28 08:55:38,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-28 08:55:38,384 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-28 08:55:38,384 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,389 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,390 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,392 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47718, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,393 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 08:55:38,394 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:41556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 08:55:38,396 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,399 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,399 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,404 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,404 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,408 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,408 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,414 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,414 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x0fb168ce to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,417 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67346c47, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,417 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,422 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,422 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,426 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,426 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,429 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,430 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,434 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,434 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x653bb8fc to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:55:38,438 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b8f652e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:55:38,444 DEBUG [hconnection-0x4bfe5239-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,444 DEBUG [hconnection-0x7e379fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,444 DEBUG [hconnection-0x7fb12004-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,445 DEBUG [hconnection-0x756a878c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,446 DEBUG [hconnection-0x492723eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,446 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,446 DEBUG [hconnection-0x3982d2fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,450 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:38,450 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47726, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,450 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,451 DEBUG [hconnection-0x2d65dc21-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,450 DEBUG [hconnection-0x70d80bf6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,452 DEBUG [hconnection-0x9a8e1f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,452 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47754, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,456 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,456 DEBUG [hconnection-0x4c6f890b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:55:38,456 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,456 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47760, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,461 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,466 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:47798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:55:38,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:38,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:38,468 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:38,468 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:38,468 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:38,468 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:38,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:38,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:38,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:38,471 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,472 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,472 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:38,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:38,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:38,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:38,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:38,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:38,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:38,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:38,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:38,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:38,476 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:38,476 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:38,476 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:38,477 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 08:55:38,487 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:38,488 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:38,488 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:38,488 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:38,488 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:38,488 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:38,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:38,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:38,496 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/82ab14360fe84d55b5283dd316e02746 2023-04-28 08:55:38,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672138502, max timestamp=1682672138509Num uniques 859; 2023-04-28 08:55:38,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:38,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672138502, max timestamp=1682672138509Num uniques 859; 2023-04-28 08:55:38,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672138502, max timestamp=1682672138509Num uniques 859; 2023-04-28 08:55:38,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:38,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:38,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672198540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1682672138535, max timestamp=1682672138559Num uniques 1018; 2023-04-28 08:55:38,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=46610, new segment data size=7680, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 08:55:38,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672198540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7681, totalHeapSize=17712, min timestamp=1682672138535, max timestamp=1682672138548Num uniques 1018; 2023-04-28 08:55:38,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=46611, new segment data size=7681, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 08:55:38,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1682672138535, max timestamp=1682672138559Num uniques 1018; 2023-04-28 08:55:38,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=46611, new segment data size=7185, suffix heap size=133840,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=157 2023-04-28 08:55:38,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672138596, max timestamp=1682672138604Num uniques 1027; 2023-04-28 08:55:38,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672138596, max timestamp=1682672138604Num uniques 1016; 2023-04-28 08:55:38,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 08:55:38,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=46536, new segment data size=10392, suffix heap size=133632,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=227 2023-04-28 08:55:38,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672138596, max timestamp=1682672138604Num uniques 1027; 2023-04-28 08:55:38,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 08:55:38,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672198641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672198642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672198642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672198675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672198675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672198746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672198746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672198746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672198885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672198885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,907 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/9d666327a74b41b4bd4d455733e39815 2023-04-28 08:55:38,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672198947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672198949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:38,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:38,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672198949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672199190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672199191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672199252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672199253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672199253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,320 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/7789ef02b0754b1fbd5673470827e9f9 2023-04-28 08:55:39,327 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/82ab14360fe84d55b5283dd316e02746 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/82ab14360fe84d55b5283dd316e02746 2023-04-28 08:55:39,331 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/82ab14360fe84d55b5283dd316e02746, entries=150, sequenceid=10, filesize=11.6 K 2023-04-28 08:55:39,333 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/9d666327a74b41b4bd4d455733e39815 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9d666327a74b41b4bd4d455733e39815 2023-04-28 08:55:39,345 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9d666327a74b41b4bd4d455733e39815, entries=150, sequenceid=10, filesize=11.6 K 2023-04-28 08:55:39,346 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/7789ef02b0754b1fbd5673470827e9f9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7789ef02b0754b1fbd5673470827e9f9 2023-04-28 08:55:39,350 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7789ef02b0754b1fbd5673470827e9f9, entries=150, sequenceid=10, filesize=11.6 K 2023-04-28 08:55:39,351 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 2ff295b65a6980314fe434234aeef4a4 in 874ms, sequenceid=10, compaction requested=false 2023-04-28 08:55:39,351 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:39,351 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:39,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:39,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:39,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:39,351 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:39,356 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:39,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:39,357 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,357 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,357 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:39,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:39,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:39,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1beafb12[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:39,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:39,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:39,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:39,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:39,360 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,360 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:39,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:39,364 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:39,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:39,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:39,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:39,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:39,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:39,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:39,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:39,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:39,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:39,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:39,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:39,371 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:39,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:39,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:39,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,374 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:39,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:39,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,375 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:39,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:39,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:39,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:39,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:39,375 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:39,375 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:39,475 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:39,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:39,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:39,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:39,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:39,480 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:39,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:39,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:39,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:39,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:39,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:39,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:39,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:39,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:39,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:39,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:39,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:39,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:39,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:39,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:39,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:39,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:39,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:39,484 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=140.89 KB heapSize=398.53 KB 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:39,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:39,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:39,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:39,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672199708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672199709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672199760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672199760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672199764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672199816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:39,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672199816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:39,901 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/b8f660d5565348eb91a4e7768848a373 2023-04-28 08:55:39,916 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/abd5cdac1698419a91159d44aec5768f 2023-04-28 08:55:40,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672200024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672200025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672200328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,332 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/76dbb612b7c345b08a86bde31c1d66a4 2023-04-28 08:55:40,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672200330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/b8f660d5565348eb91a4e7768848a373 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b8f660d5565348eb91a4e7768848a373 2023-04-28 08:55:40,339 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b8f660d5565348eb91a4e7768848a373, entries=150, sequenceid=81, filesize=11.6 K 2023-04-28 08:55:40,340 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/abd5cdac1698419a91159d44aec5768f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/abd5cdac1698419a91159d44aec5768f 2023-04-28 08:55:40,345 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/abd5cdac1698419a91159d44aec5768f, entries=150, sequenceid=81, filesize=11.6 K 2023-04-28 08:55:40,346 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/76dbb612b7c345b08a86bde31c1d66a4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/76dbb612b7c345b08a86bde31c1d66a4 2023-04-28 08:55:40,349 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/76dbb612b7c345b08a86bde31c1d66a4, entries=150, sequenceid=81, filesize=11.6 K 2023-04-28 08:55:40,350 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~398.48 KB/408048, currentSize=40.25 KB/41220 for 2ff295b65a6980314fe434234aeef4a4 in 866ms, sequenceid=81, compaction requested=false 2023-04-28 08:55:40,350 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:40,350 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:40,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:40,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:40,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:40,350 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:40,352 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:40,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,353 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:40,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,354 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:40,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cce4341[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:40,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:40,354 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:40,360 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:40,360 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:40,360 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,360 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:40,361 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:40,361 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:40,361 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,361 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:40,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:40,363 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:40,364 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:40,364 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:40,364 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:40,364 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:40,365 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,365 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,365 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:40,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:40,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:40,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:40,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:40,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:40,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,366 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:40,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:40,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:40,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:40,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:40,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:40,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:40,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:40,370 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:40,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:40,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:40,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,371 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:40,371 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:40,471 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:40,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:40,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:40,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:40,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:40,476 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:40,476 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:40,476 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:40,476 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:40,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,477 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:40,479 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:40,481 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:40,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:40,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:40,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:40,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:40,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:40,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:40,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:40,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:40,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:40,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:40,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:40,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:40,486 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:40,486 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:40,486 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:55:40,486 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:40,487 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:40,487 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:40,487 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:40,487 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:40,487 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:40,509 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/5b64c4555df440ada26da181b7e6ea05 2023-04-28 08:55:40,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:40,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682672140794, max timestamp=1682672140804Num uniques 859; 2023-04-28 08:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682672140794, max timestamp=1682672140804Num uniques 859; 2023-04-28 08:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-28 08:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682672140794, max timestamp=1682672140804Num uniques 859; 2023-04-28 08:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-28 08:55:40,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-28 08:55:40,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672200838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672200840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672200841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672200845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672200846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,947 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/60b242f5a0e04d2fad9e60ad4a23a0e7 2023-04-28 08:55:40,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672200948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672200949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,962 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672200954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:40,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672200954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,967 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/e2cccc3955774b8bb2b868fd48b27a23 2023-04-28 08:55:40,972 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/5b64c4555df440ada26da181b7e6ea05 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b64c4555df440ada26da181b7e6ea05 2023-04-28 08:55:40,977 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b64c4555df440ada26da181b7e6ea05, entries=150, sequenceid=90, filesize=11.6 K 2023-04-28 08:55:40,978 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/60b242f5a0e04d2fad9e60ad4a23a0e7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/60b242f5a0e04d2fad9e60ad4a23a0e7 2023-04-28 08:55:40,981 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/60b242f5a0e04d2fad9e60ad4a23a0e7, entries=150, sequenceid=90, filesize=11.6 K 2023-04-28 08:55:40,981 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/e2cccc3955774b8bb2b868fd48b27a23 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e2cccc3955774b8bb2b868fd48b27a23 2023-04-28 08:55:40,984 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e2cccc3955774b8bb2b868fd48b27a23, entries=150, sequenceid=90, filesize=11.6 K 2023-04-28 08:55:40,985 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 2ff295b65a6980314fe434234aeef4a4 in 499ms, sequenceid=90, compaction requested=true 2023-04-28 08:55:40,985 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:40,985 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:40,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:40,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:40,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:40,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:40,987 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,987 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,987 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:40,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:40,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ce2c5e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:40,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:40,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,990 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:40,990 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:40,990 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:40,990 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:40,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,991 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:40,994 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:40,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:40,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:40,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:40,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:40,996 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:40,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:40,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:40,996 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:40,997 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:40,998 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,998 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:40,998 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:40,999 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:41,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,001 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:41,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:41,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:41,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:41,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:41,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:41,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:41,003 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:41,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:41,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:41,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,003 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:41,003 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:41,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:41,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:41,004 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,004 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:41,004 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,104 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:41,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:41,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:41,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:41,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:41,108 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:41,108 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:41,108 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:41,108 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:41,109 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,109 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:41,110 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:41,110 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:41,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:41,110 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:41,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:41,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:41,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:41,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:41,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:41,111 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:41,111 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:41,111 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:41,112 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:41,112 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:41,112 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:41,112 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:41,121 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/20d178f16c2341a795de34ebe17a083f 2023-04-28 08:55:41,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:41,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:41,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672201167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,176 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672201169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672201173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672201174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672201277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672201277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672201279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672201286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672201483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672201484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672201485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672201493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,532 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/1c7b9ddade604942aba4c8e1b1b44e7b 2023-04-28 08:55:41,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672201788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672201789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672201789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672201801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:41,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672201851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,944 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/f4bc9669519645868e45d0c1bc6b48f8 2023-04-28 08:55:41,949 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/20d178f16c2341a795de34ebe17a083f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/20d178f16c2341a795de34ebe17a083f 2023-04-28 08:55:41,954 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/20d178f16c2341a795de34ebe17a083f, entries=150, sequenceid=126, filesize=11.6 K 2023-04-28 08:55:41,955 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/1c7b9ddade604942aba4c8e1b1b44e7b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1c7b9ddade604942aba4c8e1b1b44e7b 2023-04-28 08:55:41,958 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1c7b9ddade604942aba4c8e1b1b44e7b, entries=150, sequenceid=126, filesize=11.6 K 2023-04-28 08:55:41,959 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/f4bc9669519645868e45d0c1bc6b48f8 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f4bc9669519645868e45d0c1bc6b48f8 2023-04-28 08:55:41,962 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f4bc9669519645868e45d0c1bc6b48f8, entries=150, sequenceid=126, filesize=11.6 K 2023-04-28 08:55:41,963 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=40.25 KB/41220 for 2ff295b65a6980314fe434234aeef4a4 in 851ms, sequenceid=126, compaction requested=true 2023-04-28 08:55:41,963 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:41,963 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:41,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:41,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:41,963 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:41,966 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:41,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:41,968 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:41,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:41,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fe3b7c1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:41,970 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:41,970 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,971 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:41,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:41,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,971 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:41,971 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:41,976 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:41,977 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:41,977 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:41,977 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:41,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:41,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:41,979 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:41,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:41,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:41,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:41,979 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:41,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:41,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:41,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,982 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,983 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:41,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,986 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:41,986 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:41,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:41,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:41,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:41,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:41,987 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:41,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:41,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:41,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:41,987 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:41,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:41,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:41,987 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:41,987 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:42,087 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:42,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:42,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:42,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:42,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:42,091 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:42,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:42,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:42,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:42,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:42,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:42,093 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:42,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:42,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:42,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:42,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,093 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:42,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:42,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:42,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:42,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:42,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:42,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:42,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:42,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:42,094 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:42,094 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:42,094 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:42,095 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:55:42,095 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:42,095 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:42,095 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:42,095 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:42,095 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:42,095 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:42,104 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 08:55:42,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:42,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:42,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672142325, max timestamp=1682672142340Num uniques 859; 2023-04-28 08:55:42,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672142325, max timestamp=1682672142340Num uniques 859; 2023-04-28 08:55:42,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:42,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:42,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672142325, max timestamp=1682672142340Num uniques 859; 2023-04-28 08:55:42,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:42,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672202377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,389 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672202380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682672142351, max timestamp=1682672142387Num uniques 1018; 2023-04-28 08:55:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 08:55:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682672142351, max timestamp=1682672142387Num uniques 1018; 2023-04-28 08:55:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 08:55:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1682672142351, max timestamp=1682672142387Num uniques 1018; 2023-04-28 08:55:42,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-28 08:55:42,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672202463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672202465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672202490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672202491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,509 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/7177741b49fa4beb820ad10917e30f8a 2023-04-28 08:55:42,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672202579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672202579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672202699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672202700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672202789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:42,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672202791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:42,933 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/8c18782ea7d4409e8a5ece1ea3d72a03 2023-04-28 08:55:42,963 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/bdfefb0571d740cab050cf0961e252d8 2023-04-28 08:55:42,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/7177741b49fa4beb820ad10917e30f8a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7177741b49fa4beb820ad10917e30f8a 2023-04-28 08:55:42,977 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7177741b49fa4beb820ad10917e30f8a, entries=150, sequenceid=135, filesize=11.8 K 2023-04-28 08:55:42,981 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/8c18782ea7d4409e8a5ece1ea3d72a03 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/8c18782ea7d4409e8a5ece1ea3d72a03 2023-04-28 08:55:42,986 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/8c18782ea7d4409e8a5ece1ea3d72a03, entries=150, sequenceid=135, filesize=11.8 K 2023-04-28 08:55:42,987 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/bdfefb0571d740cab050cf0961e252d8 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/bdfefb0571d740cab050cf0961e252d8 2023-04-28 08:55:43,008 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/bdfefb0571d740cab050cf0961e252d8, entries=150, sequenceid=135, filesize=11.8 K 2023-04-28 08:55:43,009 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 2ff295b65a6980314fe434234aeef4a4 in 914ms, sequenceid=135, compaction requested=true 2023-04-28 08:55:43,009 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:43,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:43,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:43,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:43,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:43,013 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,014 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,014 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731de944[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:43,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:43,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,017 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:43,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:43,018 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:43,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:43,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:43,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,019 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-04-28 08:55:43,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:43,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:43,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:43,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:43,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:43,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:43,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:43,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:43,025 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:43,025 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:43,025 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:43,025 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:43,025 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:43,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,028 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,030 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,034 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,035 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:43,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:43,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,036 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:43,036 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:43,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/a531c90081ce4824957a1c0bfdbc6a0c 2023-04-28 08:55:43,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672203052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672203053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/c7beb9f4a1204948926b5c9f4bc690b0 2023-04-28 08:55:43,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/432c7f5c7ded4b39ae77a0c864e2e5d9 2023-04-28 08:55:43,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672203099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/a531c90081ce4824957a1c0bfdbc6a0c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a531c90081ce4824957a1c0bfdbc6a0c 2023-04-28 08:55:43,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a531c90081ce4824957a1c0bfdbc6a0c, entries=200, sequenceid=189, filesize=14.1 K 2023-04-28 08:55:43,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/c7beb9f4a1204948926b5c9f4bc690b0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/c7beb9f4a1204948926b5c9f4bc690b0 2023-04-28 08:55:43,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672203103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/c7beb9f4a1204948926b5c9f4bc690b0, entries=150, sequenceid=189, filesize=11.8 K 2023-04-28 08:55:43,118 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/432c7f5c7ded4b39ae77a0c864e2e5d9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/432c7f5c7ded4b39ae77a0c864e2e5d9 2023-04-28 08:55:43,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/432c7f5c7ded4b39ae77a0c864e2e5d9, entries=150, sequenceid=189, filesize=11.8 K 2023-04-28 08:55:43,129 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 2ff295b65a6980314fe434234aeef4a4 in 107ms, sequenceid=189, compaction requested=true 2023-04-28 08:55:43,129 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:43,129 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:43,129 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:55:43,129 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:43,129 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:55:43,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:43,131 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74006 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:55:43,131 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:43,131 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71616 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:55:43,132 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,132 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:43,132 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/82ab14360fe84d55b5283dd316e02746, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b8f660d5565348eb91a4e7768848a373, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b64c4555df440ada26da181b7e6ea05, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/20d178f16c2341a795de34ebe17a083f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7177741b49fa4beb820ad10917e30f8a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a531c90081ce4824957a1c0bfdbc6a0c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=72.3 K 2023-04-28 08:55:43,132 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,132 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9d666327a74b41b4bd4d455733e39815, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/abd5cdac1698419a91159d44aec5768f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/60b242f5a0e04d2fad9e60ad4a23a0e7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1c7b9ddade604942aba4c8e1b1b44e7b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/8c18782ea7d4409e8a5ece1ea3d72a03, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/c7beb9f4a1204948926b5c9f4bc690b0] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=69.9 K 2023-04-28 08:55:43,132 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 82ab14360fe84d55b5283dd316e02746, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682672138456 2023-04-28 08:55:43,132 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9d666327a74b41b4bd4d455733e39815, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682672138456 2023-04-28 08:55:43,133 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b8f660d5565348eb91a4e7768848a373, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682672138640 2023-04-28 08:55:43,133 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting abd5cdac1698419a91159d44aec5768f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682672138640 2023-04-28 08:55:43,133 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 5b64c4555df440ada26da181b7e6ea05, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1682672139699 2023-04-28 08:55:43,133 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 60b242f5a0e04d2fad9e60ad4a23a0e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1682672139699 2023-04-28 08:55:43,134 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 20d178f16c2341a795de34ebe17a083f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682672140839 2023-04-28 08:55:43,134 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 1c7b9ddade604942aba4c8e1b1b44e7b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682672140839 2023-04-28 08:55:43,134 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7177741b49fa4beb820ad10917e30f8a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682672141159 2023-04-28 08:55:43,134 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8c18782ea7d4409e8a5ece1ea3d72a03, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682672141159 2023-04-28 08:55:43,134 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a531c90081ce4824957a1c0bfdbc6a0c, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682672142447 2023-04-28 08:55:43,135 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c7beb9f4a1204948926b5c9f4bc690b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682672142461 2023-04-28 08:55:43,136 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:43,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:43,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:43,166 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#416 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:43,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:43,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:43,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:43,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:43,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:43,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:43,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:43,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:43,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:43,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:43,178 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:43,178 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,178 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:43,178 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:43,182 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,182 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,183 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:43,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:43,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:43,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:43,200 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:43,200 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,200 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:43,212 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#417 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:43,219 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:43,219 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:43,221 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,221 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:43,221 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:43,221 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:43,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:43,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:43,244 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:43,245 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,245 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,246 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d57682d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:43,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:43,252 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,253 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:43,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:43,254 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:43,254 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:43,256 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:43,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,257 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:43,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:43,259 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:43,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:43,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:43,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:43,259 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:43,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,260 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,261 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,266 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,266 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:43,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:43,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:43,266 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:43,266 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:43,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672143246, max timestamp=1682672143302Num uniques 859; 2023-04-28 08:55:43,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672143246, max timestamp=1682672143302Num uniques 859; 2023-04-28 08:55:43,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:43,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672143246, max timestamp=1682672143302Num uniques 859; 2023-04-28 08:55:43,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:43,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:43,367 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:43,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:43,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:43,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:43,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:43,379 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:43,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:43,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:43,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,380 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:43,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:43,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:43,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:43,383 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:43,383 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,383 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:43,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:43,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:43,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:43,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:43,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:43,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:43,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:43,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:43,387 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,387 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:43,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65f31505[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:43,409 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:43,409 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,411 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,411 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:43,411 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:43,411 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:43,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:43,412 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:43,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:43,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:43,419 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:43,419 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:43,419 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:43,419 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:43,419 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:43,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,430 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,434 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:43,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:43,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,435 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:43,435 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:43,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682672143375, max timestamp=1682672143445Num uniques 1018; 2023-04-28 08:55:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682672143375, max timestamp=1682672143445Num uniques 1018; 2023-04-28 08:55:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 08:55:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 08:55:43,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682672143375, max timestamp=1682672143445Num uniques 1018; 2023-04-28 08:55:43,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 08:55:43,535 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:43,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:43,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:43,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672203527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:43,543 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:43,543 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,543 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:43,543 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:43,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:43,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:43,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:43,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:43,546 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:43,546 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,546 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:43,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:43,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:43,547 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,547 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:43,547 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:43,547 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:43,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:43,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:43,547 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:43,548 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,548 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,548 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672203532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:43,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@478cfbc5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:43,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:43,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:43,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:43,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:43,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:43,556 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:43,556 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,556 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:43,556 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:43,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:43,559 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:43,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:43,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:43,559 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:43,559 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:43,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,566 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,567 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:43,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:43,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,574 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:43,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,574 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:43,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672203610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672203616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672203643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,660 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/677346c055844c3ca3d887d311729063 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/677346c055844c3ca3d887d311729063 2023-04-28 08:55:43,664 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into 677346c055844c3ca3d887d311729063(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:43,665 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:43,665 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=10, startTime=1682672143129; duration=0sec 2023-04-28 08:55:43,665 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:43,665 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:55:43,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672203651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,667 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71616 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:55:43,667 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:43,667 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,667 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7789ef02b0754b1fbd5673470827e9f9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/76dbb612b7c345b08a86bde31c1d66a4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e2cccc3955774b8bb2b868fd48b27a23, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f4bc9669519645868e45d0c1bc6b48f8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/bdfefb0571d740cab050cf0961e252d8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/432c7f5c7ded4b39ae77a0c864e2e5d9] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=69.9 K 2023-04-28 08:55:43,667 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7789ef02b0754b1fbd5673470827e9f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1682672138456 2023-04-28 08:55:43,668 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 76dbb612b7c345b08a86bde31c1d66a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1682672138640 2023-04-28 08:55:43,668 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e2cccc3955774b8bb2b868fd48b27a23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1682672139699 2023-04-28 08:55:43,669 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f4bc9669519645868e45d0c1bc6b48f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1682672140839 2023-04-28 08:55:43,670 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bdfefb0571d740cab050cf0961e252d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1682672141159 2023-04-28 08:55:43,670 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 432c7f5c7ded4b39ae77a0c864e2e5d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682672142461 2023-04-28 08:55:43,674 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:43,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:43,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:43,702 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#419 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:43,707 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:43,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/7ff1a72d3c844a84944bab10a5e769b5 2023-04-28 08:55:43,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:43,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:43,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,708 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,710 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:43,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:43,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:43,718 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/1660d23ef1514d9dbbc6d3786b01ded7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1660d23ef1514d9dbbc6d3786b01ded7 2023-04-28 08:55:43,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:43,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:43,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:43,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:43,725 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 1660d23ef1514d9dbbc6d3786b01ded7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:43,725 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:43,725 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=10, startTime=1682672143129; duration=0sec 2023-04-28 08:55:43,726 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:43,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:43,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:43,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:43,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:43,743 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:43,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:43,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,752 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:43,752 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:43,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,754 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:43,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc76db9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:43,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:43,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,757 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:43,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:43,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:43,758 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,758 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:43,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:43,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:43,764 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:43,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:43,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:43,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:43,764 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:43,766 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,766 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,766 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:43,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:43,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,773 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:43,773 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672203859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,873 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:43,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:43,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672203866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:43,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672203867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:43,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:43,882 DEBUG [Thread-3113] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:43,883 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:43,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:43,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:43,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,885 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:43,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:43,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:43,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:43,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:43,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:43,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:43,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:43,892 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,892 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:43,892 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:43,892 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:43,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:43,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:43,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:43,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:43,894 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,898 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,898 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:43,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d0770ec[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:43,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:43,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:43,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:43,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:43,902 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,902 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:43,903 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:43,903 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,903 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:43,903 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:43,907 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:43,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:43,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:43,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:43,907 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:43,909 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,909 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,909 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:43,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:43,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,912 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,918 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:43,918 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:43,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:43,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:43,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:43,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:43,920 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,921 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,921 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:43,921 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:43,921 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,921 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:43,921 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:43,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:43,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:43,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:43,921 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:43,921 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:44,021 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:44,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:44,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:44,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:44,029 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:44,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:44,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:44,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,030 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:44,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:44,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:44,035 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:44,035 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,035 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:44,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:44,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:44,043 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,043 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:44,043 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:44,043 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:44,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:44,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:44,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:44,046 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,046 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:44,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a10774[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:44,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:44,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,051 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:44,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:44,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:44,052 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,052 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:44,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:44,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:44,054 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:44,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:44,059 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:44,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:44,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:44,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:44,059 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:44,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,064 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,067 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:44,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:44,072 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:44,072 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:44,078 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,172 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/19c79da6f07347028537e98cb5508cc4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/19c79da6f07347028537e98cb5508cc4 2023-04-28 08:55:44,172 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:44,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:44,179 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 19c79da6f07347028537e98cb5508cc4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:44,179 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:44,179 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=10, startTime=1682672143130; duration=0sec 2023-04-28 08:55:44,179 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:44,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672204168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:44,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672204181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:44,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:44,190 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:44,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:44,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:44,191 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,191 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:44,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:44,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:44,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:44,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:44,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:44,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:44,209 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,209 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:44,209 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:44,209 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:44,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:44,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:44,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:44,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/4ac7aee9403746ba952b55372b872d45 2023-04-28 08:55:44,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,213 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,214 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,214 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:44,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d38876[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:44,216 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:44,216 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,217 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,218 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:44,218 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:44,218 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:44,218 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,218 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:44,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:44,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:44,221 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:44,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:44,224 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:44,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:44,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:44,224 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:44,224 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:44,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,228 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,229 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,230 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,237 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,237 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,238 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,238 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,238 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:44,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:44,238 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,238 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,239 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,239 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,239 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:44,239 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:44,239 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/c2f5115933564b14ab9428686563bf61 2023-04-28 08:55:44,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/7ff1a72d3c844a84944bab10a5e769b5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7ff1a72d3c844a84944bab10a5e769b5 2023-04-28 08:55:44,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7ff1a72d3c844a84944bab10a5e769b5, entries=250, sequenceid=199, filesize=16.4 K 2023-04-28 08:55:44,280 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/4ac7aee9403746ba952b55372b872d45 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4ac7aee9403746ba952b55372b872d45 2023-04-28 08:55:44,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4ac7aee9403746ba952b55372b872d45, entries=150, sequenceid=199, filesize=11.8 K 2023-04-28 08:55:44,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/c2f5115933564b14ab9428686563bf61 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c2f5115933564b14ab9428686563bf61 2023-04-28 08:55:44,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c2f5115933564b14ab9428686563bf61, entries=150, sequenceid=199, filesize=11.8 K 2023-04-28 08:55:44,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ff295b65a6980314fe434234aeef4a4 in 1125ms, sequenceid=199, compaction requested=false 2023-04-28 08:55:44,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:44,339 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:44,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:44,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:44,343 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:44,343 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,343 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:44,343 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:44,343 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,343 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,345 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,347 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:44,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:44,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:44,348 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:44,348 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,348 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:44,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:44,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:44,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:44,363 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-28 08:55:44,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:44,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:44,364 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:44,364 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:44,364 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:44,364 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:44,428 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/b6642c4036564eb3b9d89e9b4c65a4be 2023-04-28 08:55:44,473 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/ac340fd6f6a04cc499cc5e246b03998b 2023-04-28 08:55:44,520 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/576b1c0a232b4fcfb5cf728e50cd8fa2 2023-04-28 08:55:44,529 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/b6642c4036564eb3b9d89e9b4c65a4be as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b6642c4036564eb3b9d89e9b4c65a4be 2023-04-28 08:55:44,538 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b6642c4036564eb3b9d89e9b4c65a4be, entries=150, sequenceid=255, filesize=11.8 K 2023-04-28 08:55:44,539 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/ac340fd6f6a04cc499cc5e246b03998b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/ac340fd6f6a04cc499cc5e246b03998b 2023-04-28 08:55:44,547 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/ac340fd6f6a04cc499cc5e246b03998b, entries=150, sequenceid=255, filesize=11.8 K 2023-04-28 08:55:44,550 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/576b1c0a232b4fcfb5cf728e50cd8fa2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/576b1c0a232b4fcfb5cf728e50cd8fa2 2023-04-28 08:55:44,556 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/576b1c0a232b4fcfb5cf728e50cd8fa2, entries=150, sequenceid=255, filesize=11.8 K 2023-04-28 08:55:44,557 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=0 B/0 for 2ff295b65a6980314fe434234aeef4a4 in 194ms, sequenceid=255, compaction requested=true 2023-04-28 08:55:44,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:44,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:44,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:44,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:44,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:44,560 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:44,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30198d70[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:44,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:44,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:44,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:44,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:44,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:44,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,571 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:44,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:44,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:44,573 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:44,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:44,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:44,573 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:44,573 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:44,580 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,580 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,583 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,584 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,587 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:44,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,587 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,587 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,640 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:44,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:44,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:44,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:44,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:44,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:44,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:44,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:44,687 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:44,688 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/99761ee611be46f98e4c616dd969a566 2023-04-28 08:55:44,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:44,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:44,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:44,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:44,699 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:44,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:44,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:44,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,703 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:44,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672144685, max timestamp=1682672144709Num uniques 859; 2023-04-28 08:55:44,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:44,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672144685, max timestamp=1682672144709Num uniques 859; 2023-04-28 08:55:44,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:44,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672144685, max timestamp=1682672144709Num uniques 859; 2023-04-28 08:55:44,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:44,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:44,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:44,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:44,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,719 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:44,727 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:44,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:44,728 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,728 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:44,728 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:44,728 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:44,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:44,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:44,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:44,732 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,732 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,733 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,733 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:44,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62f1b8f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:44,736 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:44,736 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,737 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:44,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:44,738 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:44,738 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,738 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:44,739 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:44,739 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,739 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,739 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:44,741 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:44,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:44,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:44,742 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:44,742 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:44,743 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,743 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,746 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,749 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,754 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:44,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:44,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,755 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:44,755 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:44,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672204752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672204752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672204762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672204768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,855 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:44,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:44,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:44,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:44,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:44,867 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:44,867 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,867 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:44,867 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:44,867 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,868 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:44,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:44,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:44,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:44,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:44,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:44,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:44,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672204870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672204870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672204873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:44,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:44,887 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,889 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:44,889 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:44,889 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:44,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:44,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:44,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:44,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:44,891 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,893 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ecd3699[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:44,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:44,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,895 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:44,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:44,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:44,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:44,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672204882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,898 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:44,898 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,898 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:44,898 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:44,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:44,901 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:44,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:44,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:44,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:44,901 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:44,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:44,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:44,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,905 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:44,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:44,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:44,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:44,908 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:44,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:44,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:44,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:44,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:44,909 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:44,909 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:44,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:45,009 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:45,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:45,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:45,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:45,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:45,022 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:45,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:45,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:45,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,026 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,026 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,026 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:45,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:45,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:45,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:45,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:45,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,031 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:45,032 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:45,032 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:45,032 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,032 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:45,033 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:45,034 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,036 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:45,036 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:45,036 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,036 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:45,036 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:45,040 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:45,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:45,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3291810e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:45,042 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:45,042 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,043 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:45,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:45,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:45,044 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,044 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:45,045 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:45,045 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,045 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:45,045 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:45,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:45,059 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:45,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:45,060 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:45,060 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:45,060 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:45,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,065 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,070 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,071 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:45,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:45,071 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:45,071 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:45,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672205082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672205083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672205083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672205099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/9221fb359508489bb9aabc8e828301d1 2023-04-28 08:55:45,149 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/2dbeb15e52a044fdb78daec071ec19fd 2023-04-28 08:55:45,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/99761ee611be46f98e4c616dd969a566 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/99761ee611be46f98e4c616dd969a566 2023-04-28 08:55:45,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/99761ee611be46f98e4c616dd969a566, entries=200, sequenceid=265, filesize=14.3 K 2023-04-28 08:55:45,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/9221fb359508489bb9aabc8e828301d1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9221fb359508489bb9aabc8e828301d1 2023-04-28 08:55:45,171 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:45,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:45,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:45,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9221fb359508489bb9aabc8e828301d1, entries=150, sequenceid=265, filesize=11.9 K 2023-04-28 08:55:45,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/2dbeb15e52a044fdb78daec071ec19fd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/2dbeb15e52a044fdb78daec071ec19fd 2023-04-28 08:55:45,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:45,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:45,188 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:45,188 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,188 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:45,188 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:45,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/2dbeb15e52a044fdb78daec071ec19fd, entries=150, sequenceid=265, filesize=11.9 K 2023-04-28 08:55:45,189 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,189 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ff295b65a6980314fe434234aeef4a4 in 549ms, sequenceid=265, compaction requested=true 2023-04-28 08:55:45,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:45,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:45,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:45,190 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:45,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 08:55:45,190 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:45,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,191 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,191 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,191 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48500 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:45,191 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:45,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,191 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,192 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55720 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:45,192 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1660d23ef1514d9dbbc6d3786b01ded7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4ac7aee9403746ba952b55372b872d45, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/ac340fd6f6a04cc499cc5e246b03998b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9221fb359508489bb9aabc8e828301d1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=47.4 K 2023-04-28 08:55:45,192 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:45,192 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,192 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/677346c055844c3ca3d887d311729063, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7ff1a72d3c844a84944bab10a5e769b5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b6642c4036564eb3b9d89e9b4c65a4be, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/99761ee611be46f98e4c616dd969a566] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=54.4 K 2023-04-28 08:55:45,192 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1660d23ef1514d9dbbc6d3786b01ded7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682672142461 2023-04-28 08:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:45,192 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 677346c055844c3ca3d887d311729063, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682672142461 2023-04-28 08:55:45,192 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4ac7aee9403746ba952b55372b872d45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682672143051 2023-04-28 08:55:45,193 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7ff1a72d3c844a84944bab10a5e769b5, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682672143041 2023-04-28 08:55:45,193 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ac340fd6f6a04cc499cc5e246b03998b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682672143503 2023-04-28 08:55:45,193 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b6642c4036564eb3b9d89e9b4c65a4be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682672143503 2023-04-28 08:55:45,193 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9221fb359508489bb9aabc8e828301d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672144632 2023-04-28 08:55:45,193 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 99761ee611be46f98e4c616dd969a566, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672144632 2023-04-28 08:55:45,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:45,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:45,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:45,220 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#428 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:45,233 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:45,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:45,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:45,269 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,269 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:45,269 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 08:55:45,271 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:45,271 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:45,271 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:45,271 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:45,271 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:45,271 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:45,386 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/4be67934964c4753a44960b31ee4b366 2023-04-28 08:55:45,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:45,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:45,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672205422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47758 deadline: 1682672205422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672205423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,441 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/6a60090e453248cf8d87cbf5ae5fc8f0 2023-04-28 08:55:45,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672205428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,469 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/3d313fb7389846d7a6d21d6f8dd41d2f 2023-04-28 08:55:45,473 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/4be67934964c4753a44960b31ee4b366 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4be67934964c4753a44960b31ee4b366 2023-04-28 08:55:45,481 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4be67934964c4753a44960b31ee4b366, entries=150, sequenceid=300, filesize=11.9 K 2023-04-28 08:55:45,482 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/6a60090e453248cf8d87cbf5ae5fc8f0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6a60090e453248cf8d87cbf5ae5fc8f0 2023-04-28 08:55:45,496 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6a60090e453248cf8d87cbf5ae5fc8f0, entries=150, sequenceid=300, filesize=11.9 K 2023-04-28 08:55:45,498 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/3d313fb7389846d7a6d21d6f8dd41d2f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/3d313fb7389846d7a6d21d6f8dd41d2f 2023-04-28 08:55:45,516 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/3d313fb7389846d7a6d21d6f8dd41d2f, entries=150, sequenceid=300, filesize=11.9 K 2023-04-28 08:55:45,517 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 2ff295b65a6980314fe434234aeef4a4 in 248ms, sequenceid=300, compaction requested=true 2023-04-28 08:55:45,517 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:45,517 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:45,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:45,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:45,518 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:45,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:45,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e1ff821[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:45,524 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:45,524 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,530 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,530 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:45,530 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:45,530 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:45,531 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,531 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:45,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:45,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:45,532 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:45,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:45,534 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:45,534 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:45,534 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:45,534 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:45,534 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:45,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:45,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:55:45,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:45,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:45,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:45,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:45,541 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,541 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,543 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,550 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:45,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:45,551 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:45,551 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:45,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/0d7d9b7bd8304542ab9570ca23985523 2023-04-28 08:55:45,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672145587, max timestamp=1682672145609Num uniques 859; 2023-04-28 08:55:45,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:45,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672145587, max timestamp=1682672145609Num uniques 859; 2023-04-28 08:55:45,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:45,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672145587, max timestamp=1682672145610Num uniques 859; 2023-04-28 08:55:45,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:45,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/534c4446c1d3483d8a03c81833f61749 2023-04-28 08:55:45,651 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:45,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:45,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:45,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:45,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:45,656 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:45,656 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,656 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:45,656 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:45,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,659 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,659 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,661 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:45,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:45,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:45,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:45,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:45,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:45,666 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:45,667 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,667 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:45,667 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:45,667 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:45,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:45,668 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,668 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:45,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3012710e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:45,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:45,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:45,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:45,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:45,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:45,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:45,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,673 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:45,678 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:45,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:45,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:45,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:45,678 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:45,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:45,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:45,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,687 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:45,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672205675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,687 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:45,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672205676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672205677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672205678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,706 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/0b81167743fb4a06a7d4102641232133 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0b81167743fb4a06a7d4102641232133 2023-04-28 08:55:45,711 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into 0b81167743fb4a06a7d4102641232133(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:45,711 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:45,711 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=12, startTime=1682672145190; duration=0sec 2023-04-28 08:55:45,711 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:45,711 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:55:45,715 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60686 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:55:45,715 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:45,715 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,716 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/19c79da6f07347028537e98cb5508cc4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c2f5115933564b14ab9428686563bf61, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/576b1c0a232b4fcfb5cf728e50cd8fa2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/2dbeb15e52a044fdb78daec071ec19fd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/3d313fb7389846d7a6d21d6f8dd41d2f] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=59.3 K 2023-04-28 08:55:45,716 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 19c79da6f07347028537e98cb5508cc4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1682672142461 2023-04-28 08:55:45,716 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c2f5115933564b14ab9428686563bf61, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1682672143051 2023-04-28 08:55:45,717 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 576b1c0a232b4fcfb5cf728e50cd8fa2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1682672143503 2023-04-28 08:55:45,717 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2dbeb15e52a044fdb78daec071ec19fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672144632 2023-04-28 08:55:45,717 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3d313fb7389846d7a6d21d6f8dd41d2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682672144747 2023-04-28 08:55:45,735 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#436 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:45,757 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/1de1aed1bfab4fe2b45cee56f5b7269b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1de1aed1bfab4fe2b45cee56f5b7269b 2023-04-28 08:55:45,762 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 1de1aed1bfab4fe2b45cee56f5b7269b(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:45,762 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:45,762 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=12, startTime=1682672145190; duration=0sec 2023-04-28 08:55:45,762 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:45,785 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/fd8b3782f28646f9913442d42938c4ae as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/fd8b3782f28646f9913442d42938c4ae 2023-04-28 08:55:45,787 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:45,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:45,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:45,792 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into fd8b3782f28646f9913442d42938c4ae(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:45,792 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:45,792 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=11, startTime=1682672145190; duration=0sec 2023-04-28 08:55:45,792 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:45,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672205788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672205788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:45,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:45,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672205789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,795 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:45,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:45,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:45,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,798 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,798 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:45,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:45,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:45,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:45,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:45,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:45,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:45,802 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:45,802 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:45,802 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,802 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:45,803 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:45,803 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,803 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:45,803 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:45,803 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,803 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:45,803 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:45,804 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:45,806 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672205794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27fcf06b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:45,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:45,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:45,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:45,809 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:45,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:45,810 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:45,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:45,813 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:45,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:45,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:45,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:45,813 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:45,814 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,814 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,814 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,815 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,818 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,820 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,821 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,821 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:45,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:45,821 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,821 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,821 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:45,821 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:45,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:45,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,921 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:45,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:45,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:45,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:45,927 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:45,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:45,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:45,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,930 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,930 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,931 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,931 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:45,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:45,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:45,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:45,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:45,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:45,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:45,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:45,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,932 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:45,933 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:45,933 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:45,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:45,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:45,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:45,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:45,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,935 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,936 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,936 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:45,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19db8c6b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:45,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:45,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,940 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:45,940 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:45,940 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:45,941 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,941 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:45,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:45,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,945 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:45,946 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:45,956 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:45,956 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:45,956 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:45,956 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:45,956 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:45,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:45,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:45,962 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:45,965 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:45,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:45,967 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:45,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:45,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:45,967 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:45,968 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:45,968 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:45,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:45,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:46,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672205997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672205997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,004 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47756 deadline: 1682672205997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672206008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/018647ea27cd43cca158fb97ddba535d 2023-04-28 08:55:46,068 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:46,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:46,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:46,074 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:46,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:46,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:46,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:46,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,074 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/0d7d9b7bd8304542ab9570ca23985523 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0d7d9b7bd8304542ab9570ca23985523 2023-04-28 08:55:46,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,076 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:46,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:46,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:46,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:46,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:46,078 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:46,078 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:46,078 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:46,078 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:46,078 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:46,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0d7d9b7bd8304542ab9570ca23985523, entries=200, sequenceid=311, filesize=14.3 K 2023-04-28 08:55:46,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/534c4446c1d3483d8a03c81833f61749 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/534c4446c1d3483d8a03c81833f61749 2023-04-28 08:55:46,083 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:46,084 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:46,084 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,084 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:46,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/534c4446c1d3483d8a03c81833f61749, entries=150, sequenceid=311, filesize=11.9 K 2023-04-28 08:55:46,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:46,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2834a181[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:46,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:46,086 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:46,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:46,088 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:46,088 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,088 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:46,089 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,089 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:46,090 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:46,090 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:46,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:46,092 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:46,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:46,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:46,092 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:46,092 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:46,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:46,094 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/018647ea27cd43cca158fb97ddba535d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/018647ea27cd43cca158fb97ddba535d 2023-04-28 08:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:46,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,098 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:46,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,102 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:46,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:46,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:46,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:46,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:46,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,103 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:46,103 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:46,102 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:46,103 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,103 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:46,109 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/018647ea27cd43cca158fb97ddba535d, entries=150, sequenceid=311, filesize=11.9 K 2023-04-28 08:55:46,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 2ff295b65a6980314fe434234aeef4a4 in 577ms, sequenceid=311, compaction requested=true 2023-04-28 08:55:46,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:46,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:46,115 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:46,115 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:46,115 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:46,116 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:46,121 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:46,121 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:46,121 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:46,121 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0b81167743fb4a06a7d4102641232133, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4be67934964c4753a44960b31ee4b366, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0d7d9b7bd8304542ab9570ca23985523] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=38.4 K 2023-04-28 08:55:46,121 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 0b81167743fb4a06a7d4102641232133, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672144632 2023-04-28 08:55:46,122 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4be67934964c4753a44960b31ee4b366, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682672144747 2023-04-28 08:55:46,122 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:46,122 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:46,122 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:46,122 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1de1aed1bfab4fe2b45cee56f5b7269b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6a60090e453248cf8d87cbf5ae5fc8f0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/534c4446c1d3483d8a03c81833f61749] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.0 K 2023-04-28 08:55:46,122 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 1de1aed1bfab4fe2b45cee56f5b7269b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1682672144632 2023-04-28 08:55:46,123 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 0d7d9b7bd8304542ab9570ca23985523, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682672145405 2023-04-28 08:55:46,123 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6a60090e453248cf8d87cbf5ae5fc8f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682672144747 2023-04-28 08:55:46,123 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 534c4446c1d3483d8a03c81833f61749, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682672145405 2023-04-28 08:55:46,133 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:46,141 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:46,179 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/394ec33ad2af431f8466dcb0d160651b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/394ec33ad2af431f8466dcb0d160651b 2023-04-28 08:55:46,182 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/ff2f48831e724ccd8321c0575754e748 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ff2f48831e724ccd8321c0575754e748 2023-04-28 08:55:46,185 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 394ec33ad2af431f8466dcb0d160651b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:46,186 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:46,186 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=13, startTime=1682672146115; duration=0sec 2023-04-28 08:55:46,186 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:46,186 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 08:55:46,187 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:55:46,187 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:55:46,187 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. because compaction request was cancelled 2023-04-28 08:55:46,188 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into ff2f48831e724ccd8321c0575754e748(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:46,188 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:46,188 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672146115; duration=0sec 2023-04-28 08:55:46,188 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:46,203 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:46,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:46,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:46,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:46,206 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:46,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:46,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:46,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:46,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,206 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,207 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:46,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:46,207 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:46,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:46,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:46,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:46,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:46,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:46,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:46,209 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:46,209 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:46,209 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:46,210 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-28 08:55:46,210 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:46,210 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:46,210 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:46,210 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:46,210 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:46,210 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:46,220 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/a3c0132a98e1434db6fff1693e313526 2023-04-28 08:55:46,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:46,309 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:46,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672206321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672206322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672206323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672206324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672206433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672206434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672206434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672206434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,631 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/540479f61dc145bdbd7b9d2b8722fc33 2023-04-28 08:55:46,650 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/732a1bfcdea340298fde326b7b063acf 2023-04-28 08:55:46,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47760 deadline: 1682672206644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47742 deadline: 1682672206644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672206645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:46,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672206645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,654 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/a3c0132a98e1434db6fff1693e313526 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a3c0132a98e1434db6fff1693e313526 2023-04-28 08:55:46,657 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a3c0132a98e1434db6fff1693e313526, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 08:55:46,658 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/540479f61dc145bdbd7b9d2b8722fc33 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/540479f61dc145bdbd7b9d2b8722fc33 2023-04-28 08:55:46,661 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/540479f61dc145bdbd7b9d2b8722fc33, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 08:55:46,662 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/732a1bfcdea340298fde326b7b063acf as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/732a1bfcdea340298fde326b7b063acf 2023-04-28 08:55:46,668 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/732a1bfcdea340298fde326b7b063acf, entries=150, sequenceid=352, filesize=11.9 K 2023-04-28 08:55:46,669 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2ff295b65a6980314fe434234aeef4a4 in 459ms, sequenceid=352, compaction requested=true 2023-04-28 08:55:46,669 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:46,669 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:46,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:46,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:46,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:46,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:46,671 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:46,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:46,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:46,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0d3f88[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:46,674 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:46,674 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,677 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,677 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:46,677 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:46,677 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:46,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:46,679 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:46,679 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:46,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,679 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:46,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:46,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:46,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:46,684 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:46,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:46,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:46,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:46,684 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:46,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:46,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:46,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:46,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,687 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:46,689 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:46,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:46,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:46,690 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:46,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:46,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:46,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:46,690 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:46,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:46,791 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:46,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:46,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:46,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:46,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:46,794 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:46,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:46,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:46,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:46,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,796 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:46,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:46,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,798 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:46,798 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:46,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:46,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:46,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:46,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:46,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:46,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:46,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:46,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:46,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:46,807 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:46,807 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:46,827 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/de9ed13639d0497fa5875f3757b2f75d 2023-04-28 08:55:46,862 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/42edfa104a5a4d65a327b15328c454a2 2023-04-28 08:55:46,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:46,957 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:46,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672146959, max timestamp=1682672146991Num uniques 859; 2023-04-28 08:55:46,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:46,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672146959, max timestamp=1682672146991Num uniques 859; 2023-04-28 08:55:46,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:46,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672146959, max timestamp=1682672146991Num uniques 859; 2023-04-28 08:55:46,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:47,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672207020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672207022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672207029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672207030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672207131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672207132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672207132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672207141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,275 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/e7393f532a564cd1afce2b99dde3fa9c 2023-04-28 08:55:47,279 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/de9ed13639d0497fa5875f3757b2f75d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/de9ed13639d0497fa5875f3757b2f75d 2023-04-28 08:55:47,282 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/de9ed13639d0497fa5875f3757b2f75d, entries=150, sequenceid=362, filesize=11.9 K 2023-04-28 08:55:47,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/42edfa104a5a4d65a327b15328c454a2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/42edfa104a5a4d65a327b15328c454a2 2023-04-28 08:55:47,286 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/42edfa104a5a4d65a327b15328c454a2, entries=150, sequenceid=362, filesize=11.9 K 2023-04-28 08:55:47,286 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/e7393f532a564cd1afce2b99dde3fa9c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e7393f532a564cd1afce2b99dde3fa9c 2023-04-28 08:55:47,293 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e7393f532a564cd1afce2b99dde3fa9c, entries=150, sequenceid=362, filesize=11.9 K 2023-04-28 08:55:47,293 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ff295b65a6980314fe434234aeef4a4 in 486ms, sequenceid=362, compaction requested=true 2023-04-28 08:55:47,293 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:47,293 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:47,293 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:47,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:47,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:47,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b32f92a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:47,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:47,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:47,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:47,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:47,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:47,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:47,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:47,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:47,300 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:47,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:47,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:47,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:47,300 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:47,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,308 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,310 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,310 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,310 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:47,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:47,310 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,311 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:47,311 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:47,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:47,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-28 08:55:47,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:47,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:47,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:47,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:47,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:47,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:47,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/bc9f3a91cfcc45749b8b7accd5c8d465 2023-04-28 08:55:47,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672207357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,373 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672207358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672207358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672207360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,411 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:47,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:47,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:47,417 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:47,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:47,418 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:47,418 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,418 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,419 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,419 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:47,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:47,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:47,421 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:47,421 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,421 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:47,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:47,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:47,422 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,422 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:47,422 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:47,422 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:47,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:47,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:47,422 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:47,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,425 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,426 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:47,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e4f8cf3[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:47,429 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:47,429 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,431 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:47,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:47,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:47,431 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,431 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:47,433 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:47,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,433 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:47,436 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:47,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:47,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:47,436 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:47,436 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:47,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,446 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:47,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:47,447 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:47,447 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:47,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672207469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672207475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672207475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672207475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,547 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:47,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:47,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:47,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:47,550 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:47,550 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,550 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:47,550 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:47,551 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,551 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,553 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,554 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,554 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:47,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:47,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:47,555 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,555 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:47,555 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:47,555 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:47,555 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:47,556 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa1c12[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:47,558 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:47,558 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,561 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:47,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:47,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:47,561 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:47,562 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,562 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,562 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,562 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:47,564 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:47,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:47,564 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:47,565 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:47,565 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:47,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:47,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:47,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,576 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:47,577 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,677 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:47,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:47,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:47,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:47,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:47,680 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:47,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:47,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:47,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,682 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:47,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:47,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:47,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:47,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:47,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:47,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:47,685 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,685 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:47,685 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:47,685 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:47,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:47,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:47,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:47,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672207679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,686 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:47,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61e9829a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:47,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:47,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:47,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:47,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:47,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,691 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:47,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:47,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672207685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:47,692 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672207685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672207685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:47,696 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:47,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:47,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:47,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:47,696 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:47,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,698 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:47,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:47,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:47,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,708 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,708 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:47,708 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:47,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,776 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/4faded1590364b7989610d840b76824c 2023-04-28 08:55:47,808 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:47,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:47,811 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:47,811 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,811 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:47,811 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:47,812 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,812 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:47,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:47,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:47,815 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:47,815 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,815 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:47,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:47,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:47,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:47,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:47,816 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:47,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:47,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:47,816 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:47,818 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,818 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,818 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,818 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d4edc24[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:47,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:47,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,823 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:47,823 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:47,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:47,823 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:47,825 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,825 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,825 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,826 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:47,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:47,828 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:47,828 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:47,828 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:47,828 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:47,828 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:47,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,830 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,831 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,833 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:47,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:47,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:47,836 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,836 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,836 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,835 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,836 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:47,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,836 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,836 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:47,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672207914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,931 DEBUG [Thread-3113] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8215 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:47,936 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:47,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:47,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:47,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:47,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:47,940 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:47,940 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,940 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:47,940 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:47,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,943 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,943 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,943 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,943 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:47,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:47,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:47,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:47,946 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,946 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:47,946 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:47,946 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,946 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:47,947 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:47,949 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,949 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,950 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:47,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f14e76a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:47,951 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:47,951 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,952 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,952 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:47,952 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:47,952 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:47,954 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:47,954 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,954 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:47,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:47,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:47,955 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:47,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:47,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:47,955 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:47,955 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:47,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,957 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:47,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:47,958 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,960 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:47,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,964 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:47,964 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:47,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:47,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:47,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:47,965 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:47,965 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:47,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:47,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672207988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672207994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:47,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:47,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672207994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672207995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,065 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:48,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:48,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:48,070 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:48,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:48,070 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:48,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,071 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,072 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:48,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:48,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:48,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,074 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:48,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:48,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:48,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:48,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:48,075 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:48,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:48,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:48,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:48,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,077 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,077 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:48,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c5d7816[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:48,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:48,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:48,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:48,080 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:48,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,080 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:48,083 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:48,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,083 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:48,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:48,086 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:48,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:48,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:48,087 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:48,087 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:48,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,088 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,089 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,094 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,095 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:48,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:48,095 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:48,095 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:48,096 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:48,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/793c574135ea4997a61940dcd1910c6c 2023-04-28 08:55:48,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/bc9f3a91cfcc45749b8b7accd5c8d465 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bc9f3a91cfcc45749b8b7accd5c8d465 2023-04-28 08:55:48,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bc9f3a91cfcc45749b8b7accd5c8d465, entries=200, sequenceid=399, filesize=14.3 K 2023-04-28 08:55:48,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/4faded1590364b7989610d840b76824c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4faded1590364b7989610d840b76824c 2023-04-28 08:55:48,196 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:48,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:48,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:48,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:48,199 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:48,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:48,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:48,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,202 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,202 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:48,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:48,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:48,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:48,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:48,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:48,203 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:48,203 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,204 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:48,204 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:48,205 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:48,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:48,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:48,205 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:48,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,207 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,208 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,208 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:48,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@318dd455[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:48,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:48,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,209 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,210 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:48,210 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:48,210 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:48,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,210 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:48,211 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:48,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,211 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4faded1590364b7989610d840b76824c, entries=150, sequenceid=399, filesize=11.9 K 2023-04-28 08:55:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/793c574135ea4997a61940dcd1910c6c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/793c574135ea4997a61940dcd1910c6c 2023-04-28 08:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:48,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:48,215 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:48,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:48,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:48,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:48,215 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:48,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,218 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/793c574135ea4997a61940dcd1910c6c, entries=150, sequenceid=399, filesize=11.9 K 2023-04-28 08:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 2ff295b65a6980314fe434234aeef4a4 in 874ms, sequenceid=399, compaction requested=true 2023-04-28 08:55:48,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 08:55:48,219 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:48,219 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,220 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:55:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,222 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51628 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:48,222 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:48,222 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,222 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ff2f48831e724ccd8321c0575754e748, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a3c0132a98e1434db6fff1693e313526, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/de9ed13639d0497fa5875f3757b2f75d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bc9f3a91cfcc45749b8b7accd5c8d465] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=50.4 K 2023-04-28 08:55:48,223 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ff2f48831e724ccd8321c0575754e748, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682672145405 2023-04-28 08:55:48,223 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61306 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:55:48,223 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:48,223 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,223 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a3c0132a98e1434db6fff1693e313526, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682672145674 2023-04-28 08:55:48,223 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/fd8b3782f28646f9913442d42938c4ae, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/018647ea27cd43cca158fb97ddba535d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/732a1bfcdea340298fde326b7b063acf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e7393f532a564cd1afce2b99dde3fa9c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/793c574135ea4997a61940dcd1910c6c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=59.9 K 2023-04-28 08:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,223 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting de9ed13639d0497fa5875f3757b2f75d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672146317 2023-04-28 08:55:48,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,223 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting fd8b3782f28646f9913442d42938c4ae, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1682672144747 2023-04-28 08:55:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,223 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bc9f3a91cfcc45749b8b7accd5c8d465, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682672147016 2023-04-28 08:55:48,224 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 018647ea27cd43cca158fb97ddba535d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682672145405 2023-04-28 08:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,224 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 732a1bfcdea340298fde326b7b063acf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682672145674 2023-04-28 08:55:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,224 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e7393f532a564cd1afce2b99dde3fa9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672146317 2023-04-28 08:55:48,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,226 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:48,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,226 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 793c574135ea4997a61940dcd1910c6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682672147016 2023-04-28 08:55:48,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:48,226 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,227 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,227 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:48,227 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:48,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,227 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,227 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,234 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:48,236 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#449 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:48,327 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:48,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:48,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:48,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:48,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:48,331 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:48,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:48,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:48,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,332 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,333 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,333 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,334 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:48,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:48,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:48,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:48,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:48,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:48,336 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:48,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,336 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:48,337 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:55:48,337 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:48,337 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:48,337 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:48,337 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:48,337 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:48,337 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:48,352 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/b9daab6096314f1aa515220dd76bbbd6 2023-04-28 08:55:48,374 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/d82048320b284da28ad88a280bbe39c3 2023-04-28 08:55:48,390 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d0c922f4ed4447b884f14f862836dde2 2023-04-28 08:55:48,402 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/b9daab6096314f1aa515220dd76bbbd6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b9daab6096314f1aa515220dd76bbbd6 2023-04-28 08:55:48,409 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b9daab6096314f1aa515220dd76bbbd6, entries=150, sequenceid=408, filesize=11.9 K 2023-04-28 08:55:48,423 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/d82048320b284da28ad88a280bbe39c3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/d82048320b284da28ad88a280bbe39c3 2023-04-28 08:55:48,434 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/d82048320b284da28ad88a280bbe39c3, entries=150, sequenceid=408, filesize=11.9 K 2023-04-28 08:55:48,435 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d0c922f4ed4447b884f14f862836dde2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d0c922f4ed4447b884f14f862836dde2 2023-04-28 08:55:48,440 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d0c922f4ed4447b884f14f862836dde2, entries=150, sequenceid=408, filesize=11.9 K 2023-04-28 08:55:48,452 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 2ff295b65a6980314fe434234aeef4a4 in 115ms, sequenceid=408, compaction requested=true 2023-04-28 08:55:48,452 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:48,452 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:48,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:48,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:48,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:48,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,454 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:48,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a6981bc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:48,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:48,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:48,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:48,458 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:48,458 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,459 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:48,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:48,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,460 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:48,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:48,462 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:48,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:48,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:48,463 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:48,463 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:48,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,466 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,467 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,467 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,470 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,470 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,470 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:48,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:48,470 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:48,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,470 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:48,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,471 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:48,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:48,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:48,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:48,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:48,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:48,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:48,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:48,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:48,533 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/8e69047faab24b2aa953e1899b7fb9c4 2023-04-28 08:55:48,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/90385840a55f4d22b343b48f31eb4909 2023-04-28 08:55:48,571 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:48,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:48,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:48,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:48,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:48,579 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:48,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:48,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:48,580 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,580 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,581 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,582 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,582 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,582 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:48,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:48,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:48,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:48,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:48,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:48,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:48,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:48,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:48,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,583 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:48,584 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:48,584 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:48,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:48,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:48,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:48,585 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:48,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ae162c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:48,587 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:48,587 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:48,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682672148538, max timestamp=1682672148577Num uniques 859; 2023-04-28 08:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682672148538, max timestamp=1682672148577Num uniques 859; 2023-04-28 08:55:48,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:48,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:48,590 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1682672148538, max timestamp=1682672148577Num uniques 859; 2023-04-28 08:55:48,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:48,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:48,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:48,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:48,595 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:48,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:48,598 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:48,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:48,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:48,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:48,598 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:48,602 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,602 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,602 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,603 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:48,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:48,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:48,606 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:48,607 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:48,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672208643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,656 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/91b866f2d3fe40eb8e0e220b2426590a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/91b866f2d3fe40eb8e0e220b2426590a 2023-04-28 08:55:48,657 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/b84d2662a425430281e4ce3518f9e8d3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b84d2662a425430281e4ce3518f9e8d3 2023-04-28 08:55:48,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672208644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672208645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672208646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,663 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 91b866f2d3fe40eb8e0e220b2426590a(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:48,663 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:48,663 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=11, startTime=1682672148219; duration=0sec 2023-04-28 08:55:48,663 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:48,663 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:55:48,664 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into b84d2662a425430281e4ce3518f9e8d3(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:48,665 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:48,665 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=12, startTime=1682672148219; duration=0sec 2023-04-28 08:55:48,665 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:48,666 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:55:48,666 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:48,666 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,667 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/394ec33ad2af431f8466dcb0d160651b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/540479f61dc145bdbd7b9d2b8722fc33, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/42edfa104a5a4d65a327b15328c454a2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4faded1590364b7989610d840b76824c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/d82048320b284da28ad88a280bbe39c3] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=59.9 K 2023-04-28 08:55:48,667 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 394ec33ad2af431f8466dcb0d160651b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1682672145405 2023-04-28 08:55:48,667 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 540479f61dc145bdbd7b9d2b8722fc33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1682672145674 2023-04-28 08:55:48,668 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 42edfa104a5a4d65a327b15328c454a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1682672146317 2023-04-28 08:55:48,668 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4faded1590364b7989610d840b76824c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682672147016 2023-04-28 08:55:48,668 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d82048320b284da28ad88a280bbe39c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1682672147356 2023-04-28 08:55:48,677 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#456 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:48,707 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:48,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:48,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:48,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:48,710 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:48,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:48,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:48,712 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,712 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,713 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,714 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:48,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:48,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:48,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:48,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:48,715 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,715 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:48,715 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:48,716 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:48,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,720 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,720 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,721 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:48,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c451a7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:48,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:48,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,726 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,726 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,726 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:48,726 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:48,727 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:48,727 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:48,728 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:48,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,728 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:48,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:48,731 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:48,731 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:48,731 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:48,731 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:48,731 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:48,734 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,734 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,735 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,737 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,739 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:48,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:48,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,742 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,742 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:48,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:48,743 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:48,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672208747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672208759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672208760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672208760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,843 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:48,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:48,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:48,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:48,846 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:48,846 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:48,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:48,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,848 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:48,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:48,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:48,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:48,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:48,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:48,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:48,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:48,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:48,853 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,853 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:48,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ab40e36[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:48,855 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:48,855 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,857 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,857 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:48,857 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:48,857 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:48,857 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,857 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:48,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:48,858 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,858 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,859 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:48,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:48,861 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:48,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:48,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:48,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:48,861 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:48,865 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,865 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,865 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,866 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,868 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,871 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:48,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,871 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,871 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:48,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:48,871 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:48,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:48,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:48,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672208953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,972 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:48,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:48,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672208968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:48,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:48,975 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:48,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:48,975 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:48,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,976 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,979 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:48,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:48,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:48,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:48,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:48,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672208968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:48,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:48,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:48,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:48,980 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:48,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672208968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/7ef28b0166404553b6e0718f06fd9337 2023-04-28 08:55:48,982 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:48,982 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:48,982 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,982 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:48,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:48,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:48,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:48,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:48,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:48,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:48,985 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,985 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,986 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,986 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/8e69047faab24b2aa953e1899b7fb9c4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8e69047faab24b2aa953e1899b7fb9c4 2023-04-28 08:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:48,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce60cbd[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:48,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:48,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,989 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:48,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:48,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:48,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:48,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:48,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:48,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:48,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:48,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8e69047faab24b2aa953e1899b7fb9c4, entries=250, sequenceid=418, filesize=16.7 K 2023-04-28 08:55:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:48,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:48,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/90385840a55f4d22b343b48f31eb4909 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/90385840a55f4d22b343b48f31eb4909 2023-04-28 08:55:49,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:49,002 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:49,002 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:49,002 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:49,002 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:49,002 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:49,005 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,005 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,005 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/90385840a55f4d22b343b48f31eb4909, entries=150, sequenceid=418, filesize=11.9 K 2023-04-28 08:55:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,005 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,006 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/7ef28b0166404553b6e0718f06fd9337 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7ef28b0166404553b6e0718f06fd9337 2023-04-28 08:55:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,007 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:49,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,010 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:49,011 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:49,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7ef28b0166404553b6e0718f06fd9337, entries=150, sequenceid=418, filesize=11.9 K 2023-04-28 08:55:49,017 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ff295b65a6980314fe434234aeef4a4 in 507ms, sequenceid=418, compaction requested=true 2023-04-28 08:55:49,017 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,017 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:49,017 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:49,017 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 08:55:49,017 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:49,018 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42018 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:49,018 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:49,018 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,018 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b84d2662a425430281e4ce3518f9e8d3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b9daab6096314f1aa515220dd76bbbd6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8e69047faab24b2aa953e1899b7fb9c4] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=41.0 K 2023-04-28 08:55:49,018 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b84d2662a425430281e4ce3518f9e8d3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682672147016 2023-04-28 08:55:49,019 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b9daab6096314f1aa515220dd76bbbd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1682672147356 2023-04-28 08:55:49,019 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8e69047faab24b2aa953e1899b7fb9c4, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682672148501 2023-04-28 08:55:49,025 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#457 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:49,052 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/6a6bcaea52734bc9a98dc53fa1ec652c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/6a6bcaea52734bc9a98dc53fa1ec652c 2023-04-28 08:55:49,058 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into 6a6bcaea52734bc9a98dc53fa1ec652c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:49,058 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,059 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672149017; duration=0sec 2023-04-28 08:55:49,059 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-28 08:55:49,059 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 5 compacting, 1 eligible, 16 blocking 2023-04-28 08:55:49,061 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:55:49,061 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:55:49,061 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. because compaction request was cancelled 2023-04-28 08:55:49,061 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:49,062 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:49,062 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:49,062 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,062 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/91b866f2d3fe40eb8e0e220b2426590a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d0c922f4ed4447b884f14f862836dde2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7ef28b0166404553b6e0718f06fd9337] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.2 K 2023-04-28 08:55:49,063 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 91b866f2d3fe40eb8e0e220b2426590a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1682672147016 2023-04-28 08:55:49,063 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting d0c922f4ed4447b884f14f862836dde2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1682672147356 2023-04-28 08:55:49,063 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7ef28b0166404553b6e0718f06fd9337, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682672148507 2023-04-28 08:55:49,069 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:49,091 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/bb26374e375144c5b34649c9be3e948b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bb26374e375144c5b34649c9be3e948b 2023-04-28 08:55:49,097 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/8b9db8f3a23e46cebe22ed1811a8e68c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8b9db8f3a23e46cebe22ed1811a8e68c 2023-04-28 08:55:49,100 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into bb26374e375144c5b34649c9be3e948b(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:49,101 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,101 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=11, startTime=1682672148219; duration=0sec 2023-04-28 08:55:49,101 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:49,102 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 8b9db8f3a23e46cebe22ed1811a8e68c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:49,103 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,103 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=13, startTime=1682672149017; duration=0sec 2023-04-28 08:55:49,103 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:49,111 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:49,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:49,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:49,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:49,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:49,119 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:49,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:49,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:49,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,120 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,121 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,121 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:49,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:49,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:49,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:49,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,122 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:49,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:49,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:49,123 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:49,123 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:49,205 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/1d4aeb4962a848b0971d83b150298704 2023-04-28 08:55:49,233 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/38cd3cd978a24618b6537eebac3335d5 2023-04-28 08:55:49,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:49,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:49,272 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=459 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d4732aa2b5ac4e718ce01accf0dbcf05 2023-04-28 08:55:49,278 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/1d4aeb4962a848b0971d83b150298704 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1d4aeb4962a848b0971d83b150298704 2023-04-28 08:55:49,286 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1d4aeb4962a848b0971d83b150298704, entries=150, sequenceid=459, filesize=11.9 K 2023-04-28 08:55:49,287 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/38cd3cd978a24618b6537eebac3335d5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/38cd3cd978a24618b6537eebac3335d5 2023-04-28 08:55:49,290 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/38cd3cd978a24618b6537eebac3335d5, entries=150, sequenceid=459, filesize=11.9 K 2023-04-28 08:55:49,292 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d4732aa2b5ac4e718ce01accf0dbcf05 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d4732aa2b5ac4e718ce01accf0dbcf05 2023-04-28 08:55:49,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672209284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672209286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,297 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d4732aa2b5ac4e718ce01accf0dbcf05, entries=150, sequenceid=459, filesize=11.9 K 2023-04-28 08:55:49,298 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=53.67 KB/54960 for 2ff295b65a6980314fe434234aeef4a4 in 174ms, sequenceid=459, compaction requested=true 2023-04-28 08:55:49,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:49,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:49,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:49,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:49,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:49,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:49,302 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:55:49,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:49,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:49,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:49,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:49,303 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:49,303 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:49,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e1b647b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:49,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:49,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,309 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,309 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,309 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:49,309 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:49,310 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:49,310 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:49,311 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,311 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:49,311 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,311 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:49,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:49,314 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:49,314 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:49,314 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:49,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:49,315 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:49,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,317 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,318 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,320 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,323 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,323 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:49,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:49,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:49,323 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,323 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,323 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:49,323 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:49,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/c0d348127c1e49a4bbf4270bc5d438d0 2023-04-28 08:55:49,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/dd67e3bbc35b42cdaf50f50936875e01 2023-04-28 08:55:49,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/0ea754ad5efc40129ea71e523174c2cb 2023-04-28 08:55:49,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/c0d348127c1e49a4bbf4270bc5d438d0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/c0d348127c1e49a4bbf4270bc5d438d0 2023-04-28 08:55:49,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/c0d348127c1e49a4bbf4270bc5d438d0, entries=200, sequenceid=471, filesize=14.3 K 2023-04-28 08:55:49,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/dd67e3bbc35b42cdaf50f50936875e01 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dd67e3bbc35b42cdaf50f50936875e01 2023-04-28 08:55:49,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dd67e3bbc35b42cdaf50f50936875e01, entries=100, sequenceid=471, filesize=9.5 K 2023-04-28 08:55:49,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/0ea754ad5efc40129ea71e523174c2cb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/0ea754ad5efc40129ea71e523174c2cb 2023-04-28 08:55:49,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672149345, max timestamp=1682672149398Num uniques 859; 2023-04-28 08:55:49,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:49,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672149345, max timestamp=1682672149398Num uniques 859; 2023-04-28 08:55:49,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/0ea754ad5efc40129ea71e523174c2cb, entries=100, sequenceid=471, filesize=9.5 K 2023-04-28 08:55:49,401 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:49,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=62.62 KB/64120 for 2ff295b65a6980314fe434234aeef4a4 in 100ms, sequenceid=471, compaction requested=true 2023-04-28 08:55:49,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:49,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:49,402 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:49,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672149345, max timestamp=1682672149398Num uniques 859; 2023-04-28 08:55:49,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:49,402 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:49,402 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:49,403 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:49,403 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=40.25 KB heapSize=101.39 KB 2023-04-28 08:55:49,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:49,404 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:49,404 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:49,404 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:49,404 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:49,404 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:49,404 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:49,404 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:49,404 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,404 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bb26374e375144c5b34649c9be3e948b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/90385840a55f4d22b343b48f31eb4909, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/38cd3cd978a24618b6537eebac3335d5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dd67e3bbc35b42cdaf50f50936875e01] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=45.8 K 2023-04-28 08:55:49,404 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39676 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:49,404 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:49,404 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,404 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bb26374e375144c5b34649c9be3e948b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1682672147356 2023-04-28 08:55:49,405 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/6a6bcaea52734bc9a98dc53fa1ec652c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1d4aeb4962a848b0971d83b150298704, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/c0d348127c1e49a4bbf4270bc5d438d0] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=38.7 K 2023-04-28 08:55:49,405 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 90385840a55f4d22b343b48f31eb4909, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682672148507 2023-04-28 08:55:49,405 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 6a6bcaea52734bc9a98dc53fa1ec652c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682672148507 2023-04-28 08:55:49,405 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 38cd3cd978a24618b6537eebac3335d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1682672148642 2023-04-28 08:55:49,406 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1d4aeb4962a848b0971d83b150298704, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1682672148642 2023-04-28 08:55:49,406 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dd67e3bbc35b42cdaf50f50936875e01, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682672149283 2023-04-28 08:55:49,406 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting c0d348127c1e49a4bbf4270bc5d438d0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682672149283 2023-04-28 08:55:49,423 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:49,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:49,428 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#466 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:49,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:49,438 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:49,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:49,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:49,439 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#467 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:49,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:49,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:49,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:49,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:49,461 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:49,461 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:49,463 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,463 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:49,463 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:49,463 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:49,463 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:49,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,465 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,465 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:49,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672149437, max timestamp=1682672149464Num uniques 859; 2023-04-28 08:55:49,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672149437, max timestamp=1682672149464Num uniques 859; 2023-04-28 08:55:49,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672149437, max timestamp=1682672149464Num uniques 859; 2023-04-28 08:55:49,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:49,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:49,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29155976[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:49,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:49,469 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,470 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,471 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:49,471 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:49,471 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:49,471 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,471 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:49,472 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,472 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:49,472 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,472 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:49,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:49,475 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:49,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:49,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:49,475 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:49,475 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:49,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,477 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:49,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:49,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,483 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,483 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,483 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,483 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,483 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:49,483 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:49,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/80718ec2d611462a93dd734f48c9d115 2023-04-28 08:55:49,492 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/9ca8f769f430499a88295bd324c6a0fc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9ca8f769f430499a88295bd324c6a0fc 2023-04-28 08:55:49,499 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/ccc984ede72c4c46ad1b9c22056db54d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ccc984ede72c4c46ad1b9c22056db54d 2023-04-28 08:55:49,502 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 9ca8f769f430499a88295bd324c6a0fc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:49,502 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,502 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=12, startTime=1682672149402; duration=0sec 2023-04-28 08:55:49,502 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:49,502 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:49,503 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34762 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:49,504 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:49,504 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,504 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8b9db8f3a23e46cebe22ed1811a8e68c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d4732aa2b5ac4e718ce01accf0dbcf05, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/0ea754ad5efc40129ea71e523174c2cb] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=33.9 K 2023-04-28 08:55:49,504 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8b9db8f3a23e46cebe22ed1811a8e68c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1682672148507 2023-04-28 08:55:49,504 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d4732aa2b5ac4e718ce01accf0dbcf05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=459, earliestPutTs=1682672148642 2023-04-28 08:55:49,505 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0ea754ad5efc40129ea71e523174c2cb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682672149283 2023-04-28 08:55:49,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/bda1395ccaa14bdab7ffedb9c1de3af9 2023-04-28 08:55:49,509 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into ccc984ede72c4c46ad1b9c22056db54d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:49,509 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,509 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672149402; duration=0sec 2023-04-28 08:55:49,509 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:49,512 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#469 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:49,523 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/87fb372e3ee249a58586c56cd286549b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/87fb372e3ee249a58586c56cd286549b 2023-04-28 08:55:49,525 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/b0448d006800485985e39f616094b9ba 2023-04-28 08:55:49,529 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 87fb372e3ee249a58586c56cd286549b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:49,530 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,530 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=13, startTime=1682672149403; duration=0sec 2023-04-28 08:55:49,530 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:49,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/80718ec2d611462a93dd734f48c9d115 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/80718ec2d611462a93dd734f48c9d115 2023-04-28 08:55:49,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/80718ec2d611462a93dd734f48c9d115, entries=300, sequenceid=493, filesize=19.0 K 2023-04-28 08:55:49,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/bda1395ccaa14bdab7ffedb9c1de3af9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bda1395ccaa14bdab7ffedb9c1de3af9 2023-04-28 08:55:49,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bda1395ccaa14bdab7ffedb9c1de3af9, entries=150, sequenceid=493, filesize=11.9 K 2023-04-28 08:55:49,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/b0448d006800485985e39f616094b9ba as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b0448d006800485985e39f616094b9ba 2023-04-28 08:55:49,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b0448d006800485985e39f616094b9ba, entries=150, sequenceid=493, filesize=11.9 K 2023-04-28 08:55:49,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~101.34 KB/103776, currentSize=147.60 KB/151140 for 2ff295b65a6980314fe434234aeef4a4 in 141ms, sequenceid=493, compaction requested=false 2023-04-28 08:55:49,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:49,544 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=154.31 KB heapSize=436.08 KB 2023-04-28 08:55:49,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:49,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:55:49,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:49,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:55:49,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:49,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:55:49,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672209547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672209556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672209557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,583 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:49,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:49,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:49,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:49,586 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:49,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:49,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:49,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,587 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:49,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:49,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:49,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:49,597 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:49,597 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,598 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:49,598 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:49,599 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,599 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,599 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,599 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:49,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:49,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@334a37dc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:49,601 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:49,601 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,603 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,604 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:49,604 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:49,604 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:49,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:49,605 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:49,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,605 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,605 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:49,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:49,607 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:49,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:49,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:49,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:49,607 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:49,609 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,609 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,609 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,612 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,615 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,615 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,615 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,616 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:49,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:49,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,616 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:49,616 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:49,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:49,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672209658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672209659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672209664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,716 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:49,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:49,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:49,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:49,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:49,719 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:49,719 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,719 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:49,719 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:49,720 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,720 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,722 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:49,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:49,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:49,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:49,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:49,724 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,724 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:49,724 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:49,724 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:49,724 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:49,726 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,726 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,726 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,726 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:49,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff556ec[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:49,729 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:49,729 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,730 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:49,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:49,730 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:49,730 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,730 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:49,731 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,731 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,731 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,732 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:49,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:49,734 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:49,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:49,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:49,734 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:49,734 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:49,739 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,739 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,739 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,740 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,743 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,743 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:49,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:49,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:49,743 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:49,743 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:49,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,744 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672209799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,844 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:49,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:49,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:49,847 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:49,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:49,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:49,847 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,849 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,849 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:49,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:49,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:49,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:49,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:49,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:49,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:49,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:49,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:49,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:49,851 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:49,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:49,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:49,851 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:49,853 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,853 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,854 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:49,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:49,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af8d3d5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:49,858 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:49,858 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,860 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:49,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:49,861 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:49,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,861 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:49,862 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:49,862 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,862 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:49,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:49,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:49,867 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:49,867 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:49,867 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:49,867 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:49,867 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:49,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672209864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,869 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,869 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672209867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:49,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:49,875 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:49,875 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:49,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672209873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/4ecb115d975b466da38bff29f6836dd9 2023-04-28 08:55:49,975 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:49,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:49,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:49,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:49,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:49,978 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:49,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,978 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:49,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:49,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,979 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,981 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:49,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:49,981 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,982 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:49,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:49,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:49,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:49,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:49,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:49,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:49,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:49,983 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:49,983 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,984 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:49,984 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:49,986 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,988 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:49,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3925e315[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:49,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:49,988 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,988 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,988 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:49,989 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:49,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:49,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,989 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:49,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:49,989 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:49,991 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:49,991 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:49,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:49,993 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:49,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:49,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:49,993 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:49,993 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:49,996 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,996 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,996 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:49,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:49,997 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:50,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:50,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,000 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,000 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:50,001 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:50,101 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:50,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:50,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:50,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:50,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:50,104 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:50,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:50,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:50,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,104 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,105 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,105 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,106 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,106 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:50,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:50,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:50,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:50,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:50,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:50,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:50,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:50,107 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:50,107 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,107 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:50,107 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:50,108 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:50,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:50,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:50,108 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:50,109 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,110 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:50,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:50,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb3c73a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:50,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:50,111 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,114 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,114 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:50,114 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:50,114 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:50,114 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,114 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:50,115 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,115 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,116 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:50,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:50,118 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:50,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:50,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:50,118 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:50,118 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:50,119 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,119 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,119 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,121 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,126 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,126 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:50,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:50,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,127 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:50,127 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:50,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:50,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:50,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672210170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:50,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672210177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:50,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672210177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,227 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:50,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:50,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:50,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:50,231 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:50,231 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,231 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:50,231 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:50,231 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,231 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,232 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,232 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,232 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,232 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:50,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:50,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:50,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:50,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:50,233 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:50,233 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,233 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:50,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:50,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:50,234 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,234 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:50,234 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:50,234 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:50,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:50,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:50,234 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:50,237 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,237 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,237 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,237 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:50,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e54b896[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:50,238 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:50,239 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,240 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:50,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:50,240 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:50,240 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,240 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:50,242 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,242 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:50,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,243 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:50,245 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:50,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:50,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:50,245 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:50,246 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:50,247 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,247 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,247 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,248 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,249 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,251 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,251 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,252 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:50,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:50,252 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:50,252 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:50,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:50,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,352 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:50,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:50,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:50,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:50,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:50,356 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:50,356 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,356 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:50,356 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:50,357 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,357 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,358 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:50,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:50,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:50,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:50,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:50,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:50,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:50,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:50,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:50,363 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:50,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:50,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:50,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:50,366 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,367 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/73fcae09d3c0432b897f210dd119aef3 2023-04-28 08:55:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:50,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cebaf2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:50,369 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:50,369 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,370 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:50,370 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:50,370 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:50,372 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,372 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:50,372 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,372 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:50,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:50,374 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:50,374 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:50,374 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:50,374 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:50,374 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:50,374 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:50,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:50,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:50,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,381 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,382 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,382 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,382 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:50,382 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:50,382 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:50,482 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:50,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:50,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:50,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:50,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:50,486 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:50,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:50,486 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:50,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,488 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:50,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:50,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:50,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:50,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:50,489 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:50,490 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,490 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:50,490 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:50,490 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:50,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:50,492 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,492 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,494 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:50,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14e5ccf5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:50,496 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:50,496 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,497 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:50,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:50,497 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,497 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:50,497 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:50,504 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,504 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,504 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,505 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:50,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:50,510 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:50,511 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:50,511 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:50,511 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:50,511 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:50,512 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,512 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,512 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,514 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,518 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:50,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:50,524 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,524 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:50,524 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:50,624 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:50,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:50,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:50,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:50,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:50,628 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:50,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:50,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:50,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,629 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,630 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:50,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:50,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:50,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:50,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:50,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:50,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:50,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:50,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:50,632 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,632 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:50,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:50,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:50,633 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:50,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,634 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,636 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:50,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51aa5344[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:50,637 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:50,638 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:50,641 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:50,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:50,641 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:50,643 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:50,643 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,644 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:50,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:50,646 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:50,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:50,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:50,646 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:50,646 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:50,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,647 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,648 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,649 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,652 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:50,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,652 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,653 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:50,653 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:50,653 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,653 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:50,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:50,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672210682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:50,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672210690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:50,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672210690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,753 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:50,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:50,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:50,757 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:50,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:50,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:50,758 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,758 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,760 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,760 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,760 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,760 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:50,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:50,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:50,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:50,766 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,766 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:50,766 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:50,766 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:50,766 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:50,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:50,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a61d1ef[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:50,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:50,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,773 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:50,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:50,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:50,773 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:50,774 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,774 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:50,774 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,774 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:50,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:50,777 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:50,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:50,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:50,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:50,777 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:50,780 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,780 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,780 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:50,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:50,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:50,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,785 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:50,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:50,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,786 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:50,786 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:50,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:50,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:50,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:50,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d98de485917844e1b9c6f23929e84db9 2023-04-28 08:55:50,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/4ecb115d975b466da38bff29f6836dd9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4ecb115d975b466da38bff29f6836dd9 2023-04-28 08:55:50,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4ecb115d975b466da38bff29f6836dd9, entries=250, sequenceid=534, filesize=16.7 K 2023-04-28 08:55:50,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/73fcae09d3c0432b897f210dd119aef3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/73fcae09d3c0432b897f210dd119aef3 2023-04-28 08:55:50,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:50,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672210803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/73fcae09d3c0432b897f210dd119aef3, entries=150, sequenceid=534, filesize=11.9 K 2023-04-28 08:55:50,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d98de485917844e1b9c6f23929e84db9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d98de485917844e1b9c6f23929e84db9 2023-04-28 08:55:50,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d98de485917844e1b9c6f23929e84db9, entries=150, sequenceid=534, filesize=11.9 K 2023-04-28 08:55:50,817 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.03 KB/446496, currentSize=26.84 KB/27480 for 2ff295b65a6980314fe434234aeef4a4 in 1273ms, sequenceid=534, compaction requested=true 2023-04-28 08:55:50,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:50,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:50,817 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:50,817 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:50,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:50,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:50,818 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 49542 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:50,818 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:50,818 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:50,818 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,818 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:50,818 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ccc984ede72c4c46ad1b9c22056db54d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/80718ec2d611462a93dd734f48c9d115, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4ecb115d975b466da38bff29f6836dd9] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=48.4 K 2023-04-28 08:55:50,818 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,818 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9ca8f769f430499a88295bd324c6a0fc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bda1395ccaa14bdab7ffedb9c1de3af9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/73fcae09d3c0432b897f210dd119aef3] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.4 K 2023-04-28 08:55:50,818 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ccc984ede72c4c46ad1b9c22056db54d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682672148644 2023-04-28 08:55:50,819 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9ca8f769f430499a88295bd324c6a0fc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682672148644 2023-04-28 08:55:50,819 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 80718ec2d611462a93dd734f48c9d115, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1682672149345 2023-04-28 08:55:50,819 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bda1395ccaa14bdab7ffedb9c1de3af9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1682672149345 2023-04-28 08:55:50,819 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4ecb115d975b466da38bff29f6836dd9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672149509 2023-04-28 08:55:50,819 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 73fcae09d3c0432b897f210dd119aef3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672149529 2023-04-28 08:55:50,828 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:50,833 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:50,866 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/7b7a6dcac3224388b05582c2e563192e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7b7a6dcac3224388b05582c2e563192e 2023-04-28 08:55:50,868 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/bcf49bec3a88415dad13f75c5083ced4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bcf49bec3a88415dad13f75c5083ced4 2023-04-28 08:55:50,871 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 7b7a6dcac3224388b05582c2e563192e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:50,871 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:50,871 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=13, startTime=1682672150817; duration=0sec 2023-04-28 08:55:50,871 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:50,872 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:50,874 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into bcf49bec3a88415dad13f75c5083ced4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:50,874 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:50,874 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:50,874 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:50,874 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672150817; duration=0sec 2023-04-28 08:55:50,874 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,874 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:50,874 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/87fb372e3ee249a58586c56cd286549b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b0448d006800485985e39f616094b9ba, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d98de485917844e1b9c6f23929e84db9] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.4 K 2023-04-28 08:55:50,875 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 87fb372e3ee249a58586c56cd286549b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1682672148644 2023-04-28 08:55:50,875 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b0448d006800485985e39f616094b9ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1682672149345 2023-04-28 08:55:50,876 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d98de485917844e1b9c6f23929e84db9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672149529 2023-04-28 08:55:50,885 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#476 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:50,886 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:50,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:50,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:50,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:50,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:50,893 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:50,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:50,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:50,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,895 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:50,895 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:50,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:50,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:50,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:50,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:50,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:50,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:50,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:50,908 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:50,908 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:50,908 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:50,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:50,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:50,909 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:50,909 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:50,909 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 08:55:50,909 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:50,910 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:50,910 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:50,910 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:50,910 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:50,910 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:50,916 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/271b3285625f4d4f9e2ee1a82b212002 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/271b3285625f4d4f9e2ee1a82b212002 2023-04-28 08:55:50,923 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 271b3285625f4d4f9e2ee1a82b212002(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:50,923 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:50,923 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=13, startTime=1682672150817; duration=0sec 2023-04-28 08:55:50,924 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:50,928 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/368f1a33b0644515bd83bc68015b54f8 2023-04-28 08:55:51,343 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/3e95e1d9b214418ab375c638f3b08c23 2023-04-28 08:55:51,372 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=543 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/b174c8da109a45098ab8e4ad7f509b20 2023-04-28 08:55:51,377 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/368f1a33b0644515bd83bc68015b54f8 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/368f1a33b0644515bd83bc68015b54f8 2023-04-28 08:55:51,381 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/368f1a33b0644515bd83bc68015b54f8, entries=150, sequenceid=543, filesize=11.9 K 2023-04-28 08:55:51,382 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/3e95e1d9b214418ab375c638f3b08c23 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/3e95e1d9b214418ab375c638f3b08c23 2023-04-28 08:55:51,387 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/3e95e1d9b214418ab375c638f3b08c23, entries=150, sequenceid=543, filesize=11.9 K 2023-04-28 08:55:51,388 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/b174c8da109a45098ab8e4ad7f509b20 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b174c8da109a45098ab8e4ad7f509b20 2023-04-28 08:55:51,395 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b174c8da109a45098ab8e4ad7f509b20, entries=150, sequenceid=543, filesize=11.9 K 2023-04-28 08:55:51,396 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2ff295b65a6980314fe434234aeef4a4 in 487ms, sequenceid=543, compaction requested=false 2023-04-28 08:55:51,396 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:51,396 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:51,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:51,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:51,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:51,398 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,398 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,398 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:51,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:51,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0ff37b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:51,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:51,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:51,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:51,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:51,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:51,405 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:51,405 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,405 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,405 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:51,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:51,408 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:51,408 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:51,408 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:51,408 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:51,408 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:51,411 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,411 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,411 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,413 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,415 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:51,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,418 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,418 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,418 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:51,418 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:51,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:51,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,518 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:51,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:51,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:51,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:51,522 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:51,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:51,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:51,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:51,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,525 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:51,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:51,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:51,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:51,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:51,526 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:51,526 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:51,526 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,526 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:51,526 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:51,526 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:51,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:51,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:51,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:51,536 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47b70a4b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:51,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:51,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,539 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:51,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:51,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:51,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:51,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:51,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,541 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:51,544 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:51,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:51,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:51,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:51,544 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:51,547 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,547 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,547 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,547 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,552 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,552 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:51,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:51,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,552 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:51,552 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,653 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:51,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:51,668 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:51,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:51,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:51,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,669 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:51,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:51,672 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,672 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:51,672 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:51,672 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:51,672 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:51,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,674 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,674 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,674 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:51,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4404518a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:51,676 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:51,676 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,678 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,678 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:51,678 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:51,678 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:51,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:51,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,680 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:51,680 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,680 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:51,682 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:51,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:51,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:51,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:51,682 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:51,684 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,684 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,687 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:51,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:51,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,690 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:51,691 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:51,690 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:51,704 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:51,705 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:51,705 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:51,705 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:51,705 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:51,705 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:51,705 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:51,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/4eea18df87a14394b4a68a4ec547c04e 2023-04-28 08:55:51,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/86828a49546c42568211127b53910fd4 2023-04-28 08:55:51,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672151730, max timestamp=1682672151747Num uniques 859; 2023-04-28 08:55:51,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672151730, max timestamp=1682672151747Num uniques 859; 2023-04-28 08:55:51,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:51,749 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:51,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672151730, max timestamp=1682672151747Num uniques 859; 2023-04-28 08:55:51,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:55:51,791 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:51,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:51,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:51,794 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:51,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:51,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:51,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:51,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672211783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,797 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:51,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672211788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:51,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:51,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:51,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,799 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:51,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:51,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:51,800 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,800 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:51,800 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:51,800 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:51,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:51,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:51,800 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:51,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,801 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,802 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,802 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@406e1e4c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:51,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:51,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,804 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:51,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:51,804 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:51,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:51,805 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,806 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,806 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:51,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672211795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:51,809 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:51,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:51,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:51,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:51,809 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:51,812 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,812 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,812 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,813 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,814 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,816 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:51,817 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,817 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,817 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:51,817 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:51,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672211896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:51,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672211899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:51,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672211908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,917 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:51,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:51,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:51,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:51,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:51,921 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:51,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:51,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:51,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,923 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:51,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:51,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:51,924 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:51,925 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:51,925 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:51,925 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:51,925 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:51,926 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,926 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:51,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:51,927 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:51,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:51,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:51,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:51,927 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:51,932 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,933 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:51,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4278b911[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:51,936 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:51,936 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:51,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:51,941 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:51,941 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,941 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:51,943 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:51,943 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,943 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:51,943 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:51,948 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:51,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:51,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:51,948 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:51,948 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:51,950 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,950 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,950 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:51,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:51,951 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,954 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,958 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:51,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:51,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:51,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:51,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:51,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:51,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:51,958 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:51,958 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:51,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:51,958 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:51,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:51,959 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:51,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,059 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:52,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:52,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:52,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:52,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:52,062 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:52,062 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,062 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:52,062 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:52,063 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,063 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,064 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,065 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:52,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,066 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:52,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:52,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:52,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:52,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:52,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:52,066 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,066 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:52,066 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:52,066 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,066 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:52,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:52,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:52,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:52,068 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,068 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3de20647[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:52,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:52,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,073 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:52,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,073 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:52,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:52,073 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:52,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:52,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,075 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:52,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:52,078 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:52,078 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:52,078 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:52,078 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:52,078 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:52,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,081 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,082 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,084 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,088 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,086 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,091 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:52,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:52,091 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:52,091 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:52,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672212103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672212103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672212112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/897f873a1849402a876a089c1d5ddb4c 2023-04-28 08:55:52,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/4eea18df87a14394b4a68a4ec547c04e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4eea18df87a14394b4a68a4ec547c04e 2023-04-28 08:55:52,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4eea18df87a14394b4a68a4ec547c04e, entries=200, sequenceid=556, filesize=14.3 K 2023-04-28 08:55:52,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/86828a49546c42568211127b53910fd4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/86828a49546c42568211127b53910fd4 2023-04-28 08:55:52,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/86828a49546c42568211127b53910fd4, entries=150, sequenceid=556, filesize=11.9 K 2023-04-28 08:55:52,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/897f873a1849402a876a089c1d5ddb4c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/897f873a1849402a876a089c1d5ddb4c 2023-04-28 08:55:52,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/897f873a1849402a876a089c1d5ddb4c, entries=150, sequenceid=556, filesize=11.9 K 2023-04-28 08:55:52,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ff295b65a6980314fe434234aeef4a4 in 470ms, sequenceid=556, compaction requested=true 2023-04-28 08:55:52,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:52,174 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:52,174 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:52,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:52,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:52,177 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:52,178 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:52,178 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,178 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bcf49bec3a88415dad13f75c5083ced4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/368f1a33b0644515bd83bc68015b54f8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4eea18df87a14394b4a68a4ec547c04e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=38.9 K 2023-04-28 08:55:52,178 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:52,178 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:52,178 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,178 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7b7a6dcac3224388b05582c2e563192e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/3e95e1d9b214418ab375c638f3b08c23, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/86828a49546c42568211127b53910fd4] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.5 K 2023-04-28 08:55:52,179 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bcf49bec3a88415dad13f75c5083ced4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672149529 2023-04-28 08:55:52,179 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7b7a6dcac3224388b05582c2e563192e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672149529 2023-04-28 08:55:52,179 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 368f1a33b0644515bd83bc68015b54f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1682672149545 2023-04-28 08:55:52,179 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3e95e1d9b214418ab375c638f3b08c23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1682672149545 2023-04-28 08:55:52,179 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4eea18df87a14394b4a68a4ec547c04e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682672151695 2023-04-28 08:55:52,181 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 86828a49546c42568211127b53910fd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682672151695 2023-04-28 08:55:52,189 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:52,191 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:52,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:52,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:52,200 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:52,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:52,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:52,211 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:52,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:52,211 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:52,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,212 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,214 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,214 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,215 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:52,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:52,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:52,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:52,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:52,223 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:52,223 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,223 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:52,227 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:52,227 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:52,227 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,227 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:52,227 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-28 08:55:52,227 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:52,227 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:52,227 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:52,227 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:52,228 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:52,228 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:52,238 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/f5e38095f13047018f64ae4627a35e5b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/f5e38095f13047018f64ae4627a35e5b 2023-04-28 08:55:52,242 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into f5e38095f13047018f64ae4627a35e5b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:52,242 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,242 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672152174; duration=0sec 2023-04-28 08:55:52,242 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:52,242 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:52,245 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:52,245 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:52,245 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,245 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/271b3285625f4d4f9e2ee1a82b212002, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b174c8da109a45098ab8e4ad7f509b20, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/897f873a1849402a876a089c1d5ddb4c] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.5 K 2023-04-28 08:55:52,247 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 271b3285625f4d4f9e2ee1a82b212002, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672149529 2023-04-28 08:55:52,247 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b174c8da109a45098ab8e4ad7f509b20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=543, earliestPutTs=1682672149545 2023-04-28 08:55:52,247 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 897f873a1849402a876a089c1d5ddb4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682672151695 2023-04-28 08:55:52,256 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/267253bed5114175b4a15da843cef773 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/267253bed5114175b4a15da843cef773 2023-04-28 08:55:52,260 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 267253bed5114175b4a15da843cef773(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:52,260 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,260 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=13, startTime=1682672152174; duration=0sec 2023-04-28 08:55:52,260 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:52,271 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:52,272 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/5b97da1484144a0a9121bf7d0a4aa4fd 2023-04-28 08:55:52,296 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/4d13ad7936a841498351c14e299b5f32 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4d13ad7936a841498351c14e299b5f32 2023-04-28 08:55:52,301 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 4d13ad7936a841498351c14e299b5f32(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:52,301 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,301 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=13, startTime=1682672152174; duration=0sec 2023-04-28 08:55:52,301 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:52,309 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/de90d8afb6e54df0be21746debde9f67 2023-04-28 08:55:52,330 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d5c0179591444833bb5515cc840a6301 2023-04-28 08:55:52,335 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/5b97da1484144a0a9121bf7d0a4aa4fd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b97da1484144a0a9121bf7d0a4aa4fd 2023-04-28 08:55:52,340 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b97da1484144a0a9121bf7d0a4aa4fd, entries=150, sequenceid=591, filesize=11.9 K 2023-04-28 08:55:52,340 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/de90d8afb6e54df0be21746debde9f67 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/de90d8afb6e54df0be21746debde9f67 2023-04-28 08:55:52,348 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/de90d8afb6e54df0be21746debde9f67, entries=150, sequenceid=591, filesize=11.9 K 2023-04-28 08:55:52,350 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/d5c0179591444833bb5515cc840a6301 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d5c0179591444833bb5515cc840a6301 2023-04-28 08:55:52,355 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d5c0179591444833bb5515cc840a6301, entries=150, sequenceid=591, filesize=11.9 K 2023-04-28 08:55:52,355 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for 2ff295b65a6980314fe434234aeef4a4 in 128ms, sequenceid=591, compaction requested=false 2023-04-28 08:55:52,355 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,355 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,355 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:52,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:52,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:52,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:52,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,358 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:52,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0c8870[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:52,361 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:52,361 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,362 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,363 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,363 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:52,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:52,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:52,363 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:52,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:52,364 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:52,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:52,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:52,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:52,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:52,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:52,368 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:52,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,370 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,371 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,372 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:52,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,376 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,377 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,377 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:52,377 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:52,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:52,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:52,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:52,423 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:52,423 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:52,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:52,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:52,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:52,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:52,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/d6fa05c1bd3f469dbf9a3942db4da884 2023-04-28 08:55:52,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672152457, max timestamp=1682672152474Num uniques 859; 2023-04-28 08:55:52,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672152457, max timestamp=1682672152474Num uniques 859; 2023-04-28 08:55:52,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:52,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:52,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672152457, max timestamp=1682672152474Num uniques 859; 2023-04-28 08:55:52,477 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:52,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:55:52,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:52,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:52,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:52,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:52,481 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:52,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:52,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:52,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,481 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,482 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,483 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:52,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:52,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:52,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:52,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:52,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:52,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:52,485 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:52,485 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:52,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:52,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:52,485 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:52,486 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,488 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b206123[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:52,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:52,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:52,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:52,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:52,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,491 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:52,495 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:52,495 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,495 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:52,496 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:52,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:52,499 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:52,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:52,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:52,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:52,499 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:52,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,500 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,503 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,505 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,505 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,505 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:52,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:52,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,505 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,505 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,505 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:52,506 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:52,505 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672212517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672212525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672212528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,606 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:52,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:52,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:52,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:52,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:52,609 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:52,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:52,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:52,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,612 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,612 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:52,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:52,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:52,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:52,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:52,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:52,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:52,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:52,614 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,614 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:52,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:52,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:52,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:52,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:52,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:52,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:52,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c17fd67[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:52,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:52,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,621 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:52,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:52,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:52,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:52,623 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,623 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:52,623 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,623 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:52,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:52,626 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:52,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:52,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:52,626 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:52,626 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:52,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,632 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,634 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,635 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:52,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:52,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,635 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:52,636 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:52,636 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:52,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672212629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672212640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672212640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,736 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:52,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:52,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:52,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:52,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:52,739 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:52,740 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,740 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:52,740 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:52,740 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,740 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,741 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,741 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,742 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,742 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:52,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:52,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:52,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:52,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:52,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:52,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:52,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:52,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:52,751 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:52,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:52,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:52,751 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:52,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,753 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:52,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ea1e9c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:52,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:52,755 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:52,757 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:52,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:52,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:52,760 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:52,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:52,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:52,763 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:52,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:52,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:52,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:52,763 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:52,765 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,765 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,765 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:52,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:52,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:52,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:52,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:52,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,771 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,771 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,771 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,771 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:52,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:52,771 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:52,773 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:52,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:52,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672212809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,822 DEBUG [Thread-3105] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:52,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672212839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,852 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/dcc6db6ea60b4f249c6e311ebbc572b1 2023-04-28 08:55:52,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672212849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:52,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672212849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/a68935bd7fb34409b87f7bc65ca71e9e 2023-04-28 08:55:52,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/d6fa05c1bd3f469dbf9a3942db4da884 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/d6fa05c1bd3f469dbf9a3942db4da884 2023-04-28 08:55:52,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/d6fa05c1bd3f469dbf9a3942db4da884, entries=250, sequenceid=604, filesize=16.7 K 2023-04-28 08:55:52,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/dcc6db6ea60b4f249c6e311ebbc572b1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dcc6db6ea60b4f249c6e311ebbc572b1 2023-04-28 08:55:52,873 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:52,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:52,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:52,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:52,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:52,877 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:52,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:52,877 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:52,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dcc6db6ea60b4f249c6e311ebbc572b1, entries=150, sequenceid=604, filesize=11.9 K 2023-04-28 08:55:52,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/a68935bd7fb34409b87f7bc65ca71e9e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a68935bd7fb34409b87f7bc65ca71e9e 2023-04-28 08:55:52,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,880 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:52,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:52,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:52,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:52,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:52,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:52,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:52,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:52,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:52,881 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:52,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a68935bd7fb34409b87f7bc65ca71e9e, entries=150, sequenceid=604, filesize=11.9 K 2023-04-28 08:55:52,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 2ff295b65a6980314fe434234aeef4a4 in 461ms, sequenceid=604, compaction requested=true 2023-04-28 08:55:52,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,884 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:52,885 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:52,885 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:52,885 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:52,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:52,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:52,885 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:52,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:52,885 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-28 08:55:52,886 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:52,886 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:52,886 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:52,886 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:52,886 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:52,886 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:52,887 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42426 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:52,887 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:52,887 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:52,887 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,887 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:52,887 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/f5e38095f13047018f64ae4627a35e5b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b97da1484144a0a9121bf7d0a4aa4fd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/d6fa05c1bd3f469dbf9a3942db4da884] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=41.4 K 2023-04-28 08:55:52,887 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,887 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/267253bed5114175b4a15da843cef773, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/de90d8afb6e54df0be21746debde9f67, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dcc6db6ea60b4f249c6e311ebbc572b1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.6 K 2023-04-28 08:55:52,888 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f5e38095f13047018f64ae4627a35e5b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682672151695 2023-04-28 08:55:52,888 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 267253bed5114175b4a15da843cef773, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682672151695 2023-04-28 08:55:52,888 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 5b97da1484144a0a9121bf7d0a4aa4fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1682672151779 2023-04-28 08:55:52,889 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting de90d8afb6e54df0be21746debde9f67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1682672151779 2023-04-28 08:55:52,889 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting dcc6db6ea60b4f249c6e311ebbc572b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682672152412 2023-04-28 08:55:52,890 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting d6fa05c1bd3f469dbf9a3942db4da884, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682672152412 2023-04-28 08:55:52,900 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#493 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:52,908 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:52,932 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/01e4640d8a114dd1a81e75aea88c735b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/01e4640d8a114dd1a81e75aea88c735b 2023-04-28 08:55:52,937 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 01e4640d8a114dd1a81e75aea88c735b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:52,937 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,937 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=13, startTime=1682672152885; duration=0sec 2023-04-28 08:55:52,937 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:52,937 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:52,939 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:52,939 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:52,939 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:52,939 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4d13ad7936a841498351c14e299b5f32, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d5c0179591444833bb5515cc840a6301, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a68935bd7fb34409b87f7bc65ca71e9e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.6 K 2023-04-28 08:55:52,939 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4d13ad7936a841498351c14e299b5f32, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1682672151695 2023-04-28 08:55:52,940 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d5c0179591444833bb5515cc840a6301, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1682672151779 2023-04-28 08:55:52,940 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a68935bd7fb34409b87f7bc65ca71e9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682672152412 2023-04-28 08:55:52,941 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/ae5d27f7ee5b476b8c84eec37c7c4483 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ae5d27f7ee5b476b8c84eec37c7c4483 2023-04-28 08:55:52,948 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:52,953 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into ae5d27f7ee5b476b8c84eec37c7c4483(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:52,953 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,953 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672152884; duration=0sec 2023-04-28 08:55:52,953 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:52,966 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/df2fc9c5bce54ac085bc6a158190916c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/df2fc9c5bce54ac085bc6a158190916c 2023-04-28 08:55:52,971 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into df2fc9c5bce54ac085bc6a158190916c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:52,971 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:52,971 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=13, startTime=1682672152885; duration=0sec 2023-04-28 08:55:52,972 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:53,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:53,156 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:53,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672213167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672213168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672213173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672213274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672213274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672213281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,323 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/539979cec34f45a79adf4b5c2e739c25 2023-04-28 08:55:53,334 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/a6fc4f1f5b0a47de8d6cda764a8ed690 2023-04-28 08:55:53,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672213480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672213480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672213487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,745 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/f10ce73087e244578a3a2f3815e99d70 2023-04-28 08:55:53,749 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/539979cec34f45a79adf4b5c2e739c25 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/539979cec34f45a79adf4b5c2e739c25 2023-04-28 08:55:53,753 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/539979cec34f45a79adf4b5c2e739c25, entries=150, sequenceid=641, filesize=11.9 K 2023-04-28 08:55:53,753 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/a6fc4f1f5b0a47de8d6cda764a8ed690 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a6fc4f1f5b0a47de8d6cda764a8ed690 2023-04-28 08:55:53,756 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a6fc4f1f5b0a47de8d6cda764a8ed690, entries=150, sequenceid=641, filesize=11.9 K 2023-04-28 08:55:53,757 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/f10ce73087e244578a3a2f3815e99d70 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f10ce73087e244578a3a2f3815e99d70 2023-04-28 08:55:53,761 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f10ce73087e244578a3a2f3815e99d70, entries=150, sequenceid=641, filesize=11.9 K 2023-04-28 08:55:53,762 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 2ff295b65a6980314fe434234aeef4a4 in 877ms, sequenceid=641, compaction requested=false 2023-04-28 08:55:53,762 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:53,762 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:53,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:53,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:53,762 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,763 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:53,763 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:53,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:53,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:53,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:53,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:53,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e9a51d9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:53,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:53,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:53,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:53,768 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:53,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:53,769 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,769 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:53,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:53,770 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:53,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:53,772 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:53,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:53,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:53,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:53,772 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:53,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:53,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,778 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:53,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:53,778 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:53,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:53,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:53,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:53,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:53,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:53,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,787 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:53,787 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:53,787 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:53,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:53,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:53,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/75b6db7ae95440fbbb1fef47ab3340a3 2023-04-28 08:55:53,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672213832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672213836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672153830, max timestamp=1682672153837Num uniques 859; 2023-04-28 08:55:53,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672153830, max timestamp=1682672153837Num uniques 859; 2023-04-28 08:55:53,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:53,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672153830, max timestamp=1682672153837Num uniques 859; 2023-04-28 08:55:53,841 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:53,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:53,887 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:53,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:53,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:53,890 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:53,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:53,891 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:53,891 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:53,891 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,891 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:53,892 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:53,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:53,892 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:53,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,893 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:53,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:53,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:53,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:53,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:53,894 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:53,895 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:53,895 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:53,895 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:53,895 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:53,895 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:53,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:53,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,897 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165b0e50[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:53,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:53,899 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:53,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:53,901 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:53,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:53,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,902 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:53,902 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:53,903 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:53,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:53,905 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:53,905 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:53,905 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:53,905 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:53,905 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:53,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:53,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,908 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,908 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,909 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:53,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:53,912 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:53,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:53,913 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:53,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:53,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:53,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:53,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:53,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:53,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:53,913 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:53,913 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:53,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:53,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:53,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672213941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,953 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672213941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:53,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:53,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672213953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,013 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:54,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:54,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:54,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:54,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:54,017 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:54,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:54,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:54,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,019 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,020 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:54,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:54,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:54,021 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,021 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:54,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:54,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,021 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:54,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:54,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:54,022 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,022 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:54,022 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:54,022 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:54,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:54,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:54,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:54,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,024 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,024 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:54,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45742539[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:54,025 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:54,025 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:54,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:54,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:54,027 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,027 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:54,028 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:54,028 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,028 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:54,028 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:54,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:54,032 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:54,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:54,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:54,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:54,032 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:54,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,037 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,039 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,041 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,041 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,041 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,041 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:54,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:54,041 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,041 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,041 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,042 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:54,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,042 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,042 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,042 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,042 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:54,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672214054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672214061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,142 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:54,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:54,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:54,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:54,146 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:54,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:54,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:54,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,150 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:54,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:54,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:54,151 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:54,151 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,151 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:54,151 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:54,151 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:54,151 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,151 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:54,151 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:54,151 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:54,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:54,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:54,152 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:54,153 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,154 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547bcdf5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:54,157 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:54,157 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:54,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:54,158 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:54,159 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:54,160 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,160 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:54,160 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,160 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672214157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:54,163 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:54,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:54,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:54,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:54,164 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:54,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:54,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:54,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:54,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,173 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:54,173 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,213 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/7c8f9ffde88241d58be70e39159ed7a9 2023-04-28 08:55:54,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672214266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672214267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,273 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:54,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:54,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:54,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:54,277 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:54,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:54,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:54,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,279 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:54,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:54,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:54,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:54,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:54,281 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:54,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:54,283 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c34a3a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:54,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:54,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,287 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,288 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:54,288 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,289 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:54,289 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:54,289 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:54,290 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:54,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,290 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:54,293 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:54,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:54,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:54,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:54,293 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:54,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:54,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:54,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:54,299 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,299 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:54,299 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,400 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:54,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:54,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:54,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:54,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:54,403 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:54,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:54,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:54,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,405 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,405 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,406 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,406 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:54,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:54,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:54,407 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,407 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:54,407 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:54,407 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:54,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:54,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:54,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:54,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a4f2fc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:54,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:54,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,413 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:54,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:54,413 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:54,413 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,413 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:54,414 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:54,414 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,414 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:54,417 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:54,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:54,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:54,417 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:54,417 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:54,419 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,420 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,420 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,421 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,423 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:54,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:54,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,424 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:54,424 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:54,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672214464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,524 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:54,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:54,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:54,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:54,528 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:54,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:54,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:54,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,529 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:54,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:54,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:54,531 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,531 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:54,531 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:54,531 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:54,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:54,533 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:54,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@521d54fc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:54,534 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:54,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,538 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:54,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:54,538 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:54,538 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:54,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:54,540 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:54,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:54,542 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:54,542 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:54,542 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:54,542 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:54,542 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:54,544 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,544 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:54,544 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:54,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,546 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,547 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:54,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:54,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:54,549 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:54,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:54,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:54,550 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:54,550 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:54,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:54,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672214572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672214573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/c57bc66aa1194926a7b9500cc98d6a6f 2023-04-28 08:55:54,627 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/75b6db7ae95440fbbb1fef47ab3340a3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/75b6db7ae95440fbbb1fef47ab3340a3 2023-04-28 08:55:54,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/75b6db7ae95440fbbb1fef47ab3340a3, entries=200, sequenceid=655, filesize=14.3 K 2023-04-28 08:55:54,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/7c8f9ffde88241d58be70e39159ed7a9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7c8f9ffde88241d58be70e39159ed7a9 2023-04-28 08:55:54,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7c8f9ffde88241d58be70e39159ed7a9, entries=150, sequenceid=655, filesize=11.9 K 2023-04-28 08:55:54,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/c57bc66aa1194926a7b9500cc98d6a6f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c57bc66aa1194926a7b9500cc98d6a6f 2023-04-28 08:55:54,639 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c57bc66aa1194926a7b9500cc98d6a6f, entries=150, sequenceid=655, filesize=11.9 K 2023-04-28 08:55:54,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2ff295b65a6980314fe434234aeef4a4 in 847ms, sequenceid=655, compaction requested=true 2023-04-28 08:55:54,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:54,640 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:54,640 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:54,640 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:54,640 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:54,640 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:54,640 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:54,640 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:54,641 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:54,641 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:54,641 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,641 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,641 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ae5d27f7ee5b476b8c84eec37c7c4483, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/539979cec34f45a79adf4b5c2e739c25, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/75b6db7ae95440fbbb1fef47ab3340a3] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=39.1 K 2023-04-28 08:55:54,641 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/01e4640d8a114dd1a81e75aea88c735b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a6fc4f1f5b0a47de8d6cda764a8ed690, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7c8f9ffde88241d58be70e39159ed7a9] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.7 K 2023-04-28 08:55:54,641 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ae5d27f7ee5b476b8c84eec37c7c4483, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682672152412 2023-04-28 08:55:54,641 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 01e4640d8a114dd1a81e75aea88c735b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682672152412 2023-04-28 08:55:54,641 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 539979cec34f45a79adf4b5c2e739c25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682672152506 2023-04-28 08:55:54,641 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a6fc4f1f5b0a47de8d6cda764a8ed690, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682672152506 2023-04-28 08:55:54,642 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 75b6db7ae95440fbbb1fef47ab3340a3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682672153166 2023-04-28 08:55:54,642 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 7c8f9ffde88241d58be70e39159ed7a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682672153166 2023-04-28 08:55:54,647 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#501 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:54,650 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#502 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:54,650 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:54,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:54,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:54,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:54,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:54,655 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:54,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:54,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:54,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,664 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:54,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:54,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:54,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:54,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:54,668 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/9503bc92acd74d019e997729f60ef89e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9503bc92acd74d019e997729f60ef89e 2023-04-28 08:55:54,671 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 9503bc92acd74d019e997729f60ef89e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:54,671 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:54,672 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=13, startTime=1682672154640; duration=0sec 2023-04-28 08:55:54,672 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:54,672 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:54,672 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:54,673 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:54,673 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,673 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/df2fc9c5bce54ac085bc6a158190916c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f10ce73087e244578a3a2f3815e99d70, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c57bc66aa1194926a7b9500cc98d6a6f] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=36.7 K 2023-04-28 08:55:54,673 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting df2fc9c5bce54ac085bc6a158190916c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1682672152412 2023-04-28 08:55:54,674 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting f10ce73087e244578a3a2f3815e99d70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1682672152506 2023-04-28 08:55:54,674 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c57bc66aa1194926a7b9500cc98d6a6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682672153166 2023-04-28 08:55:54,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:54,675 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:54,675 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:54,675 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:54,682 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:54,682 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:54,683 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:54,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:54,684 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:54,705 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/4a3633bf737c4bdaacaa661b11365b37 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4a3633bf737c4bdaacaa661b11365b37 2023-04-28 08:55:54,710 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 4a3633bf737c4bdaacaa661b11365b37(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:54,710 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:54,710 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=13, startTime=1682672154640; duration=0sec 2023-04-28 08:55:54,711 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:54,713 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/3555b8d1ae254730b7e900a18f23ffee 2023-04-28 08:55:54,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:54,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:55,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:55,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672215023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,071 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/a258b8a88a2642d1b5554b0fe3f5ebd9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a258b8a88a2642d1b5554b0fe3f5ebd9 2023-04-28 08:55:55,075 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into a258b8a88a2642d1b5554b0fe3f5ebd9(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:55,075 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:55,075 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672154640; duration=0sec 2023-04-28 08:55:55,075 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:55,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672215079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672215081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672215128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,141 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/e396e490063a4bba9fe894074c9114af 2023-04-28 08:55:55,151 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=692 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/badba45e382240879ea14fa363a81b47 2023-04-28 08:55:55,155 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/3555b8d1ae254730b7e900a18f23ffee as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/3555b8d1ae254730b7e900a18f23ffee 2023-04-28 08:55:55,159 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/3555b8d1ae254730b7e900a18f23ffee, entries=150, sequenceid=692, filesize=11.9 K 2023-04-28 08:55:55,160 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/e396e490063a4bba9fe894074c9114af as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/e396e490063a4bba9fe894074c9114af 2023-04-28 08:55:55,163 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/e396e490063a4bba9fe894074c9114af, entries=150, sequenceid=692, filesize=11.9 K 2023-04-28 08:55:55,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/badba45e382240879ea14fa363a81b47 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/badba45e382240879ea14fa363a81b47 2023-04-28 08:55:55,168 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/badba45e382240879ea14fa363a81b47, entries=150, sequenceid=692, filesize=11.9 K 2023-04-28 08:55:55,169 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 2ff295b65a6980314fe434234aeef4a4 in 486ms, sequenceid=692, compaction requested=false 2023-04-28 08:55:55,169 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:55,169 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:55,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:55,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:55,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:55,169 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:55,171 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:55,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:55,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:55,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:55,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:55,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58832f2f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:55,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:55,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:55,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:55,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:55,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:55,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:55,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:55,175 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:55,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:55,176 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:55,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:55,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:55,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:55,176 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:55,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:55,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,183 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:55,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:55,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:55,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:55,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:55,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:55,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:55,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:55,188 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,188 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:55,188 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:55,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:55,288 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:55,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:55,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:55,292 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:55,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:55,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:55,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:55,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:55,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,294 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:55,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:55,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:55,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:55,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:55,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:55,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:55,295 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:55,296 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:55,296 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:55,296 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:55,296 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:55,296 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:55,296 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:55:55,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:55,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:55,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:55,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:55,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:55,297 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:55,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:55,339 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:55,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672155379, max timestamp=1682672155443Num uniques 859; 2023-04-28 08:55:55,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:55,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672155379, max timestamp=1682672155443Num uniques 859; 2023-04-28 08:55:55,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672155379, max timestamp=1682672155443Num uniques 859; 2023-04-28 08:55:55,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:55,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:55,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:55,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672215580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672215692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:55,709 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/be97f72aedd843be98fcdc33470f36ba 2023-04-28 08:55:55,719 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/a5699f72f5b2408bbe577b27abb94862 2023-04-28 08:55:55,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:55,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672215898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672216093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672216094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,140 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/a2ab5cd0fa4d4bb091afb511fe24b82f 2023-04-28 08:55:56,144 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/be97f72aedd843be98fcdc33470f36ba as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/be97f72aedd843be98fcdc33470f36ba 2023-04-28 08:55:56,147 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/be97f72aedd843be98fcdc33470f36ba, entries=150, sequenceid=705, filesize=11.9 K 2023-04-28 08:55:56,148 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/a5699f72f5b2408bbe577b27abb94862 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a5699f72f5b2408bbe577b27abb94862 2023-04-28 08:55:56,151 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a5699f72f5b2408bbe577b27abb94862, entries=150, sequenceid=705, filesize=11.9 K 2023-04-28 08:55:56,152 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/a2ab5cd0fa4d4bb091afb511fe24b82f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a2ab5cd0fa4d4bb091afb511fe24b82f 2023-04-28 08:55:56,155 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a2ab5cd0fa4d4bb091afb511fe24b82f, entries=150, sequenceid=705, filesize=11.9 K 2023-04-28 08:55:56,156 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2ff295b65a6980314fe434234aeef4a4 in 860ms, sequenceid=705, compaction requested=true 2023-04-28 08:55:56,156 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:56,156 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:56,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:56,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:56,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:56,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:56,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af33c9f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:56,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:56,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,163 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:56,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:56,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:56,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:56,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:56,164 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:56,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:56,167 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:56,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:56,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:56,167 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:56,167 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:56,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,169 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,173 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:56,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:56,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:56,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,174 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:56,174 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:56,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:56,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-28 08:55:56,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:56,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:56,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:56,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:56,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:56,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:55:56,274 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:56,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:56,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:56,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:56,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:56,277 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:56,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:56,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:56,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,280 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:56,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672216272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:56,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:56,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:56,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,281 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:56,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:56,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:56,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:56,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:56,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:56,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:56,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:56,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:56,283 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,284 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,284 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b511a2f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:56,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:56,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,286 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,286 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:56,286 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:56,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:56,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:56,287 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:56,287 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,287 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,288 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:56,291 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:56,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:56,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:56,291 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:56,291 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:56,293 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,293 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,293 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:56,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:56,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,297 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:56,298 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,298 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:56,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:56,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672216382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,398 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:56,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:56,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:56,402 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:56,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:56,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:56,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:56,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:56,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:56,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:56,405 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:56,405 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,405 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:56,406 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:56,406 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:56,406 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,407 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:56,407 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:56,407 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:56,407 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:56,408 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,409 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:56,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1353010d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:56,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:56,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,415 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,415 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:56,415 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,415 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:56,415 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:56,415 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:56,417 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,417 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:56,417 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,417 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:56,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:56,419 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:56,419 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:56,419 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:56,419 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:56,419 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:56,421 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,421 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,421 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,421 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:56,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:56,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,426 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:56,426 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:56,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,527 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:56,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:56,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:56,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:56,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:56,530 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:56,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:56,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:56,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,533 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,533 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,533 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,533 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:56,534 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:56,534 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:56,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:56,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:56,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:56,535 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:56,535 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,535 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:56,535 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:56,535 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,535 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:56,535 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:56,535 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,535 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:56,535 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:56,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,536 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:56,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5268ae1f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:56,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:56,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,539 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:56,539 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:56,540 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:56,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:56,541 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:56,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,541 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:56,543 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:56,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:56,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:56,544 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:56,544 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:56,546 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,546 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,546 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,547 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,548 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:56,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:56,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,551 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:56,551 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:56,550 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,550 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:56,551 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,551 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,551 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:56,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672216589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/9ad8b91fd24e4d68b99e5b82882892d5 2023-04-28 08:55:56,635 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/2b72b2048c7546899a7e1ea091fb7466 2023-04-28 08:55:56,651 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:56,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:56,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:56,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:56,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:56,654 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:56,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:56,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:56,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,656 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,656 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:56,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:56,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:56,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:56,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:56,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:56,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:56,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:56,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:56,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:56,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:56,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:56,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:56,660 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:56,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71dc6709[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:56,663 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:56,663 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,664 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,665 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:56,665 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:56,665 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:56,665 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,665 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:56,666 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:56,666 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,666 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,666 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:56,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:56,669 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:56,669 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:56,669 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:56,669 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:56,669 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:56,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,675 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,674 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:56,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:56,675 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,676 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,676 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:56,676 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:56,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:56,776 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:56,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:56,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:56,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:56,779 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:56,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:56,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:56,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,783 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:56,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:56,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:56,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:56,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:56,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:56,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:56,785 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,785 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:56,785 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:56,785 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:56,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:56,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:56,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:56,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,786 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:56,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527f8b7f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:56,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:56,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,789 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:56,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:56,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:56,790 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,790 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:56,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,794 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:56,794 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,794 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:56,797 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:56,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:56,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:56,797 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:56,797 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:56,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,799 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,801 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:56,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,803 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,803 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:56,804 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:56,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:56,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47756 deadline: 1682672216861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,869 DEBUG [Thread-3105] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8224 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:56,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:56,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672216897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,904 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:56,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:56,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:56,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:56,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:56,907 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:56,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:56,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:56,908 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,908 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,909 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,909 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,909 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,910 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:56,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:56,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:56,910 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:56,910 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:56,910 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:56,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:56,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:56,911 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,911 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:56,911 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:56,911 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:56,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:56,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:56,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:56,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:56,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,913 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59afb354[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:56,915 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:56,915 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,917 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,917 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:56,917 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:56,917 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:56,917 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,917 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:56,918 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:56,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:56,921 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:56,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:56,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:56,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:56,921 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:56,923 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,923 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:56,923 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:56,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:56,923 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:56,929 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:56,929 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:56,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:56,930 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:56,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:56,930 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:56,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:56,930 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:56,930 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:56,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,030 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:57,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:57,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:57,033 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:57,034 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,034 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:57,034 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:57,034 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,034 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,036 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,036 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,036 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:57,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:57,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:57,037 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:57,037 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,037 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:57,038 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:57,038 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:57,038 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,038 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:57,038 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:57,038 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,038 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:57,038 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:57,038 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,038 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:57,038 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:57,039 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,039 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,040 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:57,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e20e628[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:57,041 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:57,041 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,042 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:57,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:57,043 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:57,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:57,045 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,045 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:57,045 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,046 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:57,048 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:57,048 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:57,048 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:57,048 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:57,048 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:57,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=742 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/7d776089e6c140f5abbe453311479938 2023-04-28 08:55:57,050 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,050 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,050 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,051 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,053 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/9ad8b91fd24e4d68b99e5b82882892d5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/9ad8b91fd24e4d68b99e5b82882892d5 2023-04-28 08:55:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,056 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:57,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:57,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,057 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,057 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:57,057 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,058 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/9ad8b91fd24e4d68b99e5b82882892d5, entries=200, sequenceid=742, filesize=14.3 K 2023-04-28 08:55:57,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/2b72b2048c7546899a7e1ea091fb7466 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/2b72b2048c7546899a7e1ea091fb7466 2023-04-28 08:55:57,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/2b72b2048c7546899a7e1ea091fb7466, entries=150, sequenceid=742, filesize=11.9 K 2023-04-28 08:55:57,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/7d776089e6c140f5abbe453311479938 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7d776089e6c140f5abbe453311479938 2023-04-28 08:55:57,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7d776089e6c140f5abbe453311479938, entries=150, sequenceid=742, filesize=11.9 K 2023-04-28 08:55:57,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2ff295b65a6980314fe434234aeef4a4 in 856ms, sequenceid=742, compaction requested=true 2023-04-28 08:55:57,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,068 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:57,068 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:57,068 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:57,068 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:55:57,068 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:57,069 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52376 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:57,069 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:55:57,069 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:57,069 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:57,069 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,069 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,069 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a258b8a88a2642d1b5554b0fe3f5ebd9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/3555b8d1ae254730b7e900a18f23ffee, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/be97f72aedd843be98fcdc33470f36ba, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/9ad8b91fd24e4d68b99e5b82882892d5] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=51.1 K 2023-04-28 08:55:57,069 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9503bc92acd74d019e997729f60ef89e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/e396e490063a4bba9fe894074c9114af, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a5699f72f5b2408bbe577b27abb94862, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/2b72b2048c7546899a7e1ea091fb7466] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=48.7 K 2023-04-28 08:55:57,070 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a258b8a88a2642d1b5554b0fe3f5ebd9, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682672153166 2023-04-28 08:55:57,070 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9503bc92acd74d019e997729f60ef89e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682672153166 2023-04-28 08:55:57,070 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 3555b8d1ae254730b7e900a18f23ffee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1682672153904 2023-04-28 08:55:57,070 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e396e490063a4bba9fe894074c9114af, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1682672153904 2023-04-28 08:55:57,070 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting be97f72aedd843be98fcdc33470f36ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1682672154997 2023-04-28 08:55:57,070 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a5699f72f5b2408bbe577b27abb94862, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1682672154997 2023-04-28 08:55:57,071 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9ad8b91fd24e4d68b99e5b82882892d5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1682672155533 2023-04-28 08:55:57,071 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2b72b2048c7546899a7e1ea091fb7466, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1682672155533 2023-04-28 08:55:57,080 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#513 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:57,081 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#514 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:57,157 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:57,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:57,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:57,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:57,160 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:57,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:57,160 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:57,161 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,161 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,162 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,162 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,162 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:57,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:57,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:57,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:57,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:57,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:57,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:57,164 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:57,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:57,164 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:57,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:57,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:57,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:57,165 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:57,165 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:57,165 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:57,172 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/18e3b4374ead448ebea4c15234231ad5 2023-04-28 08:55:57,181 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/76c9b1f905a044b7ac2e9eb72e1d1100 2023-04-28 08:55:57,192 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/5a0e747f5e3641a6952797520af219ae 2023-04-28 08:55:57,200 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/18e3b4374ead448ebea4c15234231ad5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/18e3b4374ead448ebea4c15234231ad5 2023-04-28 08:55:57,205 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/18e3b4374ead448ebea4c15234231ad5, entries=150, sequenceid=752, filesize=11.9 K 2023-04-28 08:55:57,206 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/76c9b1f905a044b7ac2e9eb72e1d1100 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/76c9b1f905a044b7ac2e9eb72e1d1100 2023-04-28 08:55:57,212 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/76c9b1f905a044b7ac2e9eb72e1d1100, entries=150, sequenceid=752, filesize=11.9 K 2023-04-28 08:55:57,213 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/5a0e747f5e3641a6952797520af219ae as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/5a0e747f5e3641a6952797520af219ae 2023-04-28 08:55:57,216 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/5a0e747f5e3641a6952797520af219ae, entries=150, sequenceid=752, filesize=11.9 K 2023-04-28 08:55:57,217 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 2ff295b65a6980314fe434234aeef4a4 in 53ms, sequenceid=752, compaction requested=true 2023-04-28 08:55:57,217 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,217 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:57,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:57,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:57,217 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:57,220 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,220 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,221 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:57,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c5624ca[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:57,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:57,223 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,224 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,224 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:57,224 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:57,224 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:57,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:57,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,226 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:57,226 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:57,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:57,228 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:57,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:57,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:57,228 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:57,228 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:57,229 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,229 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,231 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,233 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,237 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,249 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:57,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:57,249 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,250 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:57,250 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:57,250 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,250 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,350 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:57,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:57,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:57,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:57,353 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:57,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:57,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:57,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,353 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,355 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,355 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,355 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,355 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:57,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:57,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:57,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:57,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:57,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,356 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:57,357 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:57,357 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:57,357 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,357 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:57,358 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,358 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:57,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:57,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:57,358 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:57,359 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,359 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,360 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,360 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:57,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72dc684c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:57,361 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:57,361 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,362 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:57,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:57,362 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,362 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:57,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:57,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:57,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,364 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,364 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:57,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:57,367 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:57,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:57,367 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:57,368 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:57,368 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:57,369 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,369 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,369 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,370 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,371 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,373 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:57,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:57,374 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:57,374 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:57,374 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,374 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,375 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:57,433 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:55:57,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:57,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:57,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:57,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:57,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:57,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:55:57,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=763 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/8ad04673b7cd44459bbac132b87995f3 2023-04-28 08:55:57,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=763 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/513e07579dcf47a4bf2fad48d800cc11 2023-04-28 08:55:57,474 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:57,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:57,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:57,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:57,478 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:57,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:57,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:57,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,478 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,480 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:57,482 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:57,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:57,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:57,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:57,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:57,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:57,483 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:57,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:57,483 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:57,484 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:57,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:57,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:57,484 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:57,486 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,486 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,487 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,488 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:57,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564d4050[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:57,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:57,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,490 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:57,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:57,490 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,490 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:57,490 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:57,493 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,493 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:57,493 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,493 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/55e3ba2805cb4ad9b07584f55d71ff5b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/55e3ba2805cb4ad9b07584f55d71ff5b 2023-04-28 08:55:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,493 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:57,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,497 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into 55e3ba2805cb4ad9b07584f55d71ff5b(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:57,497 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,497 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=12, startTime=1682672157068; duration=0sec 2023-04-28 08:55:57,497 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:57,497 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:55:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:57,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:57,498 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:57,498 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:57,498 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:57,498 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:57,498 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:57,498 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62088 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:55:57,499 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:57,499 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,499 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4a3633bf737c4bdaacaa661b11365b37, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/badba45e382240879ea14fa363a81b47, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a2ab5cd0fa4d4bb091afb511fe24b82f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7d776089e6c140f5abbe453311479938, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/5a0e747f5e3641a6952797520af219ae] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=60.6 K 2023-04-28 08:55:57,499 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 4a3633bf737c4bdaacaa661b11365b37, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1682672153166 2023-04-28 08:55:57,499 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,499 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,499 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,500 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting badba45e382240879ea14fa363a81b47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=692, earliestPutTs=1682672153904 2023-04-28 08:55:57,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,500 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a2ab5cd0fa4d4bb091afb511fe24b82f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1682672154997 2023-04-28 08:55:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,501 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,501 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7d776089e6c140f5abbe453311479938, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1682672155533 2023-04-28 08:55:57,502 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,503 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 5a0e747f5e3641a6952797520af219ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1682672156228 2023-04-28 08:55:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,505 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,506 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/15cff6bf2ec749ce8c9fdfef67d1824a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/15cff6bf2ec749ce8c9fdfef67d1824a 2023-04-28 08:55:57,506 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:57,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,506 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:57,506 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,506 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:57,511 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 15cff6bf2ec749ce8c9fdfef67d1824a(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:57,511 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,511 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=12, startTime=1682672157068; duration=0sec 2023-04-28 08:55:57,511 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:57,513 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#521 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:57,528 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/9051f12165e148799ce855f79e7dd5ac as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/9051f12165e148799ce855f79e7dd5ac 2023-04-28 08:55:57,533 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 9051f12165e148799ce855f79e7dd5ac(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:57,533 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,533 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=11, startTime=1682672157068; duration=0sec 2023-04-28 08:55:57,533 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:57,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672157521, max timestamp=1682672157569Num uniques 859; 2023-04-28 08:55:57,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672157521, max timestamp=1682672157569Num uniques 859; 2023-04-28 08:55:57,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672157521, max timestamp=1682672157569Num uniques 859; 2023-04-28 08:55:57,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:57,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:57,572 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:55:57,606 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:57,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:57,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:57,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:57,609 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:57,609 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:57,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:57,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,610 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,612 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,612 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,613 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:57,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:57,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:57,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:57,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,614 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:57,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:57,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:57,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:57,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:57,615 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:57,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:57,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:57,615 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:57,616 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,616 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,617 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:57,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:57,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f15ed17[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:57,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:57,620 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,621 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,621 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:57,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:57,622 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:57,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,622 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:57,624 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,625 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:57,625 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,625 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:57,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:57,628 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:57,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:57,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:57,628 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:57,628 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:57,629 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,630 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,631 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,632 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,637 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,638 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,639 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,639 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:57,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:57,639 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,639 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,639 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,639 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:57,639 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:57,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:57,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672217668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,739 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:57,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:57,743 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:57,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:57,743 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:57,744 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,744 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,746 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,746 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,746 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,747 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:57,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:57,748 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:57,748 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,748 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:57,748 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:57,748 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:57,748 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,748 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:57,748 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:57,748 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,748 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:57,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:57,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:57,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:57,750 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:57,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7208fa4b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:57,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:57,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,756 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:57,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:57,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:57,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:57,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:57,758 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:57,760 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:57,760 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:57,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:57,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:57,761 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:57,763 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,763 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,763 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:57,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:57,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,768 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,768 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:57,768 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:57,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:57,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672217778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=763 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/56b5c6eac46045ccaa5ad71c1447a5ea 2023-04-28 08:55:57,868 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/8ad04673b7cd44459bbac132b87995f3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8ad04673b7cd44459bbac132b87995f3 2023-04-28 08:55:57,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:57,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:57,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:57,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:57,871 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:57,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:57,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:57,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8ad04673b7cd44459bbac132b87995f3, entries=250, sequenceid=763, filesize=16.7 K 2023-04-28 08:55:57,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/513e07579dcf47a4bf2fad48d800cc11 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/513e07579dcf47a4bf2fad48d800cc11 2023-04-28 08:55:57,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,874 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:57,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:57,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:57,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:57,875 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:57,876 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,876 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:57,876 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:57,876 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:57,876 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:57,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/513e07579dcf47a4bf2fad48d800cc11, entries=150, sequenceid=763, filesize=11.9 K 2023-04-28 08:55:57,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,878 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/56b5c6eac46045ccaa5ad71c1447a5ea as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/56b5c6eac46045ccaa5ad71c1447a5ea 2023-04-28 08:55:57,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:57,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6307724e[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:57,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:57,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,882 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:57,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:57,882 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:57,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:57,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/56b5c6eac46045ccaa5ad71c1447a5ea, entries=150, sequenceid=763, filesize=11.9 K 2023-04-28 08:55:57,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2ff295b65a6980314fe434234aeef4a4 in 451ms, sequenceid=763, compaction requested=true 2023-04-28 08:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:57,884 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:57,884 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:57,884 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,884 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:57,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,885 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:57,885 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42766 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:57,885 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:55:57,885 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:55:57,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,886 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,885 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:57,886 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/55e3ba2805cb4ad9b07584f55d71ff5b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/18e3b4374ead448ebea4c15234231ad5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8ad04673b7cd44459bbac132b87995f3] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=41.8 K 2023-04-28 08:55:57,886 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/15cff6bf2ec749ce8c9fdfef67d1824a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/76c9b1f905a044b7ac2e9eb72e1d1100, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/513e07579dcf47a4bf2fad48d800cc11] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=37.0 K 2023-04-28 08:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,886 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 15cff6bf2ec749ce8c9fdfef67d1824a, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1682672155533 2023-04-28 08:55:57,886 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 55e3ba2805cb4ad9b07584f55d71ff5b, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=742, earliestPutTs=1682672155533 2023-04-28 08:55:57,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,886 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 76c9b1f905a044b7ac2e9eb72e1d1100, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1682672156228 2023-04-28 08:55:57,887 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 18e3b4374ead448ebea4c15234231ad5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1682672156228 2023-04-28 08:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,887 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 513e07579dcf47a4bf2fad48d800cc11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=763, earliestPutTs=1682672157418 2023-04-28 08:55:57,887 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8ad04673b7cd44459bbac132b87995f3, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=763, earliestPutTs=1682672157409 2023-04-28 08:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:57,887 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:57,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:57,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:57,887 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:57,887 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:57,889 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,889 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,889 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:57,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,892 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:57,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,893 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#522 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:57,893 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#523 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:57,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:57,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:57,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:57,897 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:57,897 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:57,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:57,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:57,898 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:57,898 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:57,903 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:57,903 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:57,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:57,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:57,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:57,930 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/6b46e63331714cecbdd76d08c5f3a76a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6b46e63331714cecbdd76d08c5f3a76a 2023-04-28 08:55:57,936 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into 6b46e63331714cecbdd76d08c5f3a76a(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:57,936 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:57,936 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=13, startTime=1682672157884; duration=0sec 2023-04-28 08:55:57,936 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:57,937 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 08:55:57,937 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:55:57,937 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:55:57,937 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. because compaction request was cancelled 2023-04-28 08:55:57,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:55:57,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-28 08:55:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:55:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:55:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:55:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:55:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:55:57,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:55:57,998 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:57,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:58,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:58,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:58,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:58,001 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:58,001 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,001 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:58,001 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:58,001 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,001 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,004 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:58,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:58,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:58,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:58,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:58,005 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,005 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:58,005 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:58,005 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:58,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:58,007 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,007 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,007 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:58,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff4f1be[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:58,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:58,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,009 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,009 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:58,009 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:58,009 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,009 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:58,009 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:58,011 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:58,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:58,014 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:58,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:58,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:58,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:58,014 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:58,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,017 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672218011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672218015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,020 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,020 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,020 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:58,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:58,020 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,020 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:58,020 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,020 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:58,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,021 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:58,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47760 deadline: 1682672218101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,107 DEBUG [Thread-3111] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:58,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47742 deadline: 1682672218111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,119 DEBUG [Thread-3107] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:55:58,121 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:58,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:58,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:58,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:58,124 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:58,124 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,124 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:58,124 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:58,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672218120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672218120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,128 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,128 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,129 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:58,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:58,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:58,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:58,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:58,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:58,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:58,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:58,130 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,130 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:58,130 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:58,130 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:58,130 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:58,131 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,131 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:58,131 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:58,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,133 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,134 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:58,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:58,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb9c202[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:58,136 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:58,136 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:58,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:58,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:58,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:58,139 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,139 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:58,139 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,139 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:58,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:58,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:58,145 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:58,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:58,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:58,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:58,146 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:58,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,148 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,149 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:58,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:58,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:58,152 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,153 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,153 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,153 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:58,153 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,253 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:58,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:58,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:58,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:58,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:58,257 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:58,257 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,257 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:58,257 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:58,257 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,257 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,258 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,258 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,259 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:58,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:58,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:58,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:58,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:58,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:58,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:58,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:58,260 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:58,260 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:58,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:58,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,261 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:58,261 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:58,261 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,261 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:58,261 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:58,262 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,262 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,263 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,263 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:58,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ad6a71c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:58,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:58,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:58,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:58,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:58,265 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:58,268 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,268 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:58,268 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,268 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:58,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:58,270 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:58,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:58,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:58,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:58,270 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:58,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,272 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,273 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,274 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,276 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:58,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:58,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,276 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,277 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:58,277 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:58,277 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:58,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,320 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/da6e2c2057cb43f48952285bf43c7b68 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/da6e2c2057cb43f48952285bf43c7b68 2023-04-28 08:55:58,327 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into da6e2c2057cb43f48952285bf43c7b68(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:58,327 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:58,327 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672157884; duration=0sec 2023-04-28 08:55:58,327 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:58,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672218327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672218327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,377 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:58,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:58,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:55:58,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:55:58,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:55:58,380 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:55:58,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:55:58,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:55:58,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,381 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,382 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,382 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:55:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:55:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:55:58,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:55:58,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:55:58,383 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:55:58,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. started... 2023-04-28 08:55:58,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. as already flushing 2023-04-28 08:55:58,384 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:55:58,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:55:58,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/bbec9ab4814048ff866775b308c57549 2023-04-28 08:55:58,388 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,388 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:55:58,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:55:58,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21b3b0a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:55:58,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:55:58,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,393 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:55:58,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:55:58,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:55:58,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:55:58,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:55:58,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:55:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:55:58,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:55:58,400 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:55:58,400 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:55:58,400 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:55:58,400 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:55:58,400 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:55:58,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:55:58,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:55:58,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:55:58,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,405 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:55:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:55:58,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:55:58,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:55:58,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:55:58,408 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:55:58,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:55:58,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:55:58,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:55:58,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:55:58,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:55:58,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:55:58,410 DEBUG [Thread-3115] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:55:58,410 DEBUG [Thread-3115] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:55:58,442 DEBUG [Thread-3118] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:54655 2023-04-28 08:55:58,442 DEBUG [Thread-3118] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:58,444 DEBUG [Thread-3120] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:54655 2023-04-28 08:55:58,444 DEBUG [Thread-3120] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:58,445 DEBUG [Thread-3122] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:54655 2023-04-28 08:55:58,445 DEBUG [Thread-3122] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:58,448 DEBUG [Thread-3124] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x653bb8fc to 127.0.0.1:54655 2023-04-28 08:55:58,449 DEBUG [Thread-3124] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:58,450 DEBUG [Thread-3116] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:54655 2023-04-28 08:55:58,450 DEBUG [Thread-3116] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:55:58,511 DEBUG [Thread-3115] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:55:58,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:55:58,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672218632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672218634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:58,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/28eedf282f1644db957abbdec44b1750 2023-04-28 08:55:59,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:59,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:47758 deadline: 1682672219137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:59,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:55:59,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:47770 deadline: 1682672219137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:55:59,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=805 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/44f57869087048e6ba1e1023241e3218 2023-04-28 08:55:59,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/bbec9ab4814048ff866775b308c57549 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bbec9ab4814048ff866775b308c57549 2023-04-28 08:55:59,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bbec9ab4814048ff866775b308c57549, entries=200, sequenceid=805, filesize=14.3 K 2023-04-28 08:55:59,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/28eedf282f1644db957abbdec44b1750 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/28eedf282f1644db957abbdec44b1750 2023-04-28 08:55:59,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/28eedf282f1644db957abbdec44b1750, entries=150, sequenceid=805, filesize=11.9 K 2023-04-28 08:55:59,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/44f57869087048e6ba1e1023241e3218 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/44f57869087048e6ba1e1023241e3218 2023-04-28 08:55:59,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/44f57869087048e6ba1e1023241e3218, entries=150, sequenceid=805, filesize=11.9 K 2023-04-28 08:55:59,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 2ff295b65a6980314fe434234aeef4a4 in 1263ms, sequenceid=805, compaction requested=true 2023-04-28 08:55:59,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:59,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:55:59,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:59,240 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 08:55:59,240 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 08:55:59,240 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:55:59,241 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:55:59,241 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:55:59,241 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:55:59,241 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. because compaction request was cancelled 2023-04-28 08:55:59,241 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:55:59,241 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:55:59,241 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. because compaction request was cancelled 2023-04-28 08:55:59,242 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:55:59,242 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/C is initiating minor compaction (all files) 2023-04-28 08:55:59,242 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/C in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:55:59,242 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/9051f12165e148799ce855f79e7dd5ac, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/56b5c6eac46045ccaa5ad71c1447a5ea, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/44f57869087048e6ba1e1023241e3218] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=37.0 K 2023-04-28 08:55:59,242 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9051f12165e148799ce855f79e7dd5ac, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1682672156228 2023-04-28 08:55:59,242 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 56b5c6eac46045ccaa5ad71c1447a5ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=763, earliestPutTs=1682672157418 2023-04-28 08:55:59,243 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 44f57869087048e6ba1e1023241e3218, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682672157637 2023-04-28 08:55:59,249 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#C#compaction#527 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:55:59,656 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/8d3b9df37a4a4eacbceb46550a373650 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8d3b9df37a4a4eacbceb46550a373650 2023-04-28 08:55:59,659 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/C of 2ff295b65a6980314fe434234aeef4a4 into 8d3b9df37a4a4eacbceb46550a373650(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:55:59,659 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:55:59,659 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/C, priority=13, startTime=1682672159240; duration=0sec 2023-04-28 08:55:59,660 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:00,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:56:00,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:56:00,146 DEBUG [Thread-3109] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:54655 2023-04-28 08:56:00,146 DEBUG [Thread-3109] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:00,146 DEBUG [Thread-3113] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fb168ce to 127.0.0.1:54655 2023-04-28 08:56:00,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:56:00,147 DEBUG [Thread-3113] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:00,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:00,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:56:00,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:00,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:56:00,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:00,553 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/e969bf63383941e6a817c62d16f9ed6b 2023-04-28 08:56:00,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/339d86d71303418f996dae348f0cb8f4 2023-04-28 08:56:01,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/930249343af140f8a68800b285595620 2023-04-28 08:56:01,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/e969bf63383941e6a817c62d16f9ed6b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/e969bf63383941e6a817c62d16f9ed6b 2023-04-28 08:56:01,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/e969bf63383941e6a817c62d16f9ed6b, entries=150, sequenceid=818, filesize=11.9 K 2023-04-28 08:56:01,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/339d86d71303418f996dae348f0cb8f4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/339d86d71303418f996dae348f0cb8f4 2023-04-28 08:56:01,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/339d86d71303418f996dae348f0cb8f4, entries=150, sequenceid=818, filesize=11.9 K 2023-04-28 08:56:01,381 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/930249343af140f8a68800b285595620 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/930249343af140f8a68800b285595620 2023-04-28 08:56:01,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/930249343af140f8a68800b285595620, entries=150, sequenceid=818, filesize=11.9 K 2023-04-28 08:56:01,385 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 2ff295b65a6980314fe434234aeef4a4 in 1239ms, sequenceid=818, compaction requested=true 2023-04-28 08:56:01,386 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:56:01,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:01,386 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:56:01,386 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:56:01,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:01,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:01,387 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40428 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:56:01,387 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:56:01,387 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/A is initiating minor compaction (all files) 2023-04-28 08:56:01,387 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 2ff295b65a6980314fe434234aeef4a4/B is initiating minor compaction (all files) 2023-04-28 08:56:01,387 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/A in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:56:01,387 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ff295b65a6980314fe434234aeef4a4/B in TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:56:01,387 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/da6e2c2057cb43f48952285bf43c7b68, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bbec9ab4814048ff866775b308c57549, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/e969bf63383941e6a817c62d16f9ed6b] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=39.5 K 2023-04-28 08:56:01,387 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6b46e63331714cecbdd76d08c5f3a76a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/28eedf282f1644db957abbdec44b1750, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/339d86d71303418f996dae348f0cb8f4] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp, totalSize=37.1 K 2023-04-28 08:56:01,388 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting da6e2c2057cb43f48952285bf43c7b68, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=763, earliestPutTs=1682672157418 2023-04-28 08:56:01,388 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6b46e63331714cecbdd76d08c5f3a76a, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=763, earliestPutTs=1682672157418 2023-04-28 08:56:01,388 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting bbec9ab4814048ff866775b308c57549, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682672157637 2023-04-28 08:56:01,388 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 28eedf282f1644db957abbdec44b1750, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=805, earliestPutTs=1682672157637 2023-04-28 08:56:01,388 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e969bf63383941e6a817c62d16f9ed6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1682672158002 2023-04-28 08:56:01,388 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 339d86d71303418f996dae348f0cb8f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1682672158002 2023-04-28 08:56:01,394 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#A#compaction#531 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:01,396 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ff295b65a6980314fe434234aeef4a4#B#compaction#532 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:01,802 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/1286f830fd8843db90cd7a4e93246909 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1286f830fd8843db90cd7a4e93246909 2023-04-28 08:56:01,806 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/A of 2ff295b65a6980314fe434234aeef4a4 into 1286f830fd8843db90cd7a4e93246909(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:01,806 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:56:01,806 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/A, priority=13, startTime=1682672161386; duration=0sec 2023-04-28 08:56:01,806 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:01,807 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 08:56:01,807 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:56:01,807 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:56:01,807 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. because compaction request was cancelled 2023-04-28 08:56:01,808 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/a8c4ddb8daf04b48b74c9745eb10b16f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a8c4ddb8daf04b48b74c9745eb10b16f 2023-04-28 08:56:01,811 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ff295b65a6980314fe434234aeef4a4/B of 2ff295b65a6980314fe434234aeef4a4 into a8c4ddb8daf04b48b74c9745eb10b16f(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:01,811 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:56:01,811 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4., storeName=2ff295b65a6980314fe434234aeef4a4/B, priority=13, startTime=1682672161386; duration=0sec 2023-04-28 08:56:01,811 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:02,116 DEBUG [Thread-3111] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:54655 2023-04-28 08:56:02,116 DEBUG [Thread-3111] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:02,143 DEBUG [Thread-3107] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:54655 2023-04-28 08:56:02,143 DEBUG [Thread-3107] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:06,896 DEBUG [Thread-3105] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:54655 2023-04-28 08:56:06,896 DEBUG [Thread-3105] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 89 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 257 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 159 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1600 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 4799 rows 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1585 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 4755 rows 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1583 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 4749 rows 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1589 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 4764 rows 2023-04-28 08:56:06,896 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(402): scanned 1575 2023-04-28 08:56:06,897 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(403): verified 4721 rows 2023-04-28 08:56:06,897 INFO [Listener at localhost/39717] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 08:56:06,897 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:54655 2023-04-28 08:56:06,897 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:06,898 INFO [Listener at localhost/39717] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 08:56:06,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable TestAcidGuarantees 2023-04-28 08:56:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:06,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 08:56:06,903 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672166903"}]},"ts":"1682672166903"} 2023-04-28 08:56:06,904 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 08:56:07,001 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 08:56:07,001 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ff295b65a6980314fe434234aeef4a4, UNASSIGN}] 2023-04-28 08:56:07,003 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ff295b65a6980314fe434234aeef4a4, UNASSIGN 2023-04-28 08:56:07,003 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2ff295b65a6980314fe434234aeef4a4, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:07,003 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672167003"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672167003"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672167003"}]},"ts":"1682672167003"} 2023-04-28 08:56:07,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 08:56:07,004 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:56:07,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:56:07,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 2ff295b65a6980314fe434234aeef4a4, disabling compactions & flushes 2023-04-28 08:56:07,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:56:07,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:56:07,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. after waiting 0 ms 2023-04-28 08:56:07,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:56:07,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 2ff295b65a6980314fe434234aeef4a4 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-28 08:56:07,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=A 2023-04-28 08:56:07,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:07,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=B 2023-04-28 08:56:07,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:07,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ff295b65a6980314fe434234aeef4a4, store=C 2023-04-28 08:56:07,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:07,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 08:56:07,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 08:56:07,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=826 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/498a1fe142a84f888ea8d2b01ebc7f97 2023-04-28 08:56:07,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=826 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/734f7d0fba274d4cb4af2534598c18ea 2023-04-28 08:56:08,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 08:56:08,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=826 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/4bcbc00454bd4b8cbaf8ad4ad03b7adf 2023-04-28 08:56:08,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/A/498a1fe142a84f888ea8d2b01ebc7f97 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/498a1fe142a84f888ea8d2b01ebc7f97 2023-04-28 08:56:08,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/498a1fe142a84f888ea8d2b01ebc7f97, entries=100, sequenceid=826, filesize=9.5 K 2023-04-28 08:56:08,392 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/B/734f7d0fba274d4cb4af2534598c18ea as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/734f7d0fba274d4cb4af2534598c18ea 2023-04-28 08:56:08,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/734f7d0fba274d4cb4af2534598c18ea, entries=100, sequenceid=826, filesize=9.5 K 2023-04-28 08:56:08,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/.tmp/C/4bcbc00454bd4b8cbaf8ad4ad03b7adf as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4bcbc00454bd4b8cbaf8ad4ad03b7adf 2023-04-28 08:56:08,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4bcbc00454bd4b8cbaf8ad4ad03b7adf, entries=100, sequenceid=826, filesize=9.5 K 2023-04-28 08:56:08,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 2ff295b65a6980314fe434234aeef4a4 in 1244ms, sequenceid=826, compaction requested=true 2023-04-28 08:56:08,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/82ab14360fe84d55b5283dd316e02746, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b8f660d5565348eb91a4e7768848a373, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b64c4555df440ada26da181b7e6ea05, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/20d178f16c2341a795de34ebe17a083f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7177741b49fa4beb820ad10917e30f8a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a531c90081ce4824957a1c0bfdbc6a0c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/677346c055844c3ca3d887d311729063, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7ff1a72d3c844a84944bab10a5e769b5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b6642c4036564eb3b9d89e9b4c65a4be, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/99761ee611be46f98e4c616dd969a566, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0b81167743fb4a06a7d4102641232133, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4be67934964c4753a44960b31ee4b366, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0d7d9b7bd8304542ab9570ca23985523, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ff2f48831e724ccd8321c0575754e748, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a3c0132a98e1434db6fff1693e313526, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/de9ed13639d0497fa5875f3757b2f75d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bc9f3a91cfcc45749b8b7accd5c8d465, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b84d2662a425430281e4ce3518f9e8d3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b9daab6096314f1aa515220dd76bbbd6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8e69047faab24b2aa953e1899b7fb9c4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/6a6bcaea52734bc9a98dc53fa1ec652c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1d4aeb4962a848b0971d83b150298704, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/c0d348127c1e49a4bbf4270bc5d438d0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ccc984ede72c4c46ad1b9c22056db54d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/80718ec2d611462a93dd734f48c9d115, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4ecb115d975b466da38bff29f6836dd9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bcf49bec3a88415dad13f75c5083ced4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/368f1a33b0644515bd83bc68015b54f8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4eea18df87a14394b4a68a4ec547c04e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/f5e38095f13047018f64ae4627a35e5b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b97da1484144a0a9121bf7d0a4aa4fd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/d6fa05c1bd3f469dbf9a3942db4da884, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ae5d27f7ee5b476b8c84eec37c7c4483, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/539979cec34f45a79adf4b5c2e739c25, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/75b6db7ae95440fbbb1fef47ab3340a3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a258b8a88a2642d1b5554b0fe3f5ebd9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/3555b8d1ae254730b7e900a18f23ffee, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/be97f72aedd843be98fcdc33470f36ba, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/9ad8b91fd24e4d68b99e5b82882892d5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/55e3ba2805cb4ad9b07584f55d71ff5b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/18e3b4374ead448ebea4c15234231ad5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8ad04673b7cd44459bbac132b87995f3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/da6e2c2057cb43f48952285bf43c7b68, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bbec9ab4814048ff866775b308c57549, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/e969bf63383941e6a817c62d16f9ed6b] to archive 2023-04-28 08:56:08,424 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:56:08,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/82ab14360fe84d55b5283dd316e02746 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/82ab14360fe84d55b5283dd316e02746 2023-04-28 08:56:08,427 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b8f660d5565348eb91a4e7768848a373 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b8f660d5565348eb91a4e7768848a373 2023-04-28 08:56:08,428 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b64c4555df440ada26da181b7e6ea05 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b64c4555df440ada26da181b7e6ea05 2023-04-28 08:56:08,429 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/20d178f16c2341a795de34ebe17a083f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/20d178f16c2341a795de34ebe17a083f 2023-04-28 08:56:08,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7177741b49fa4beb820ad10917e30f8a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7177741b49fa4beb820ad10917e30f8a 2023-04-28 08:56:08,431 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a531c90081ce4824957a1c0bfdbc6a0c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a531c90081ce4824957a1c0bfdbc6a0c 2023-04-28 08:56:08,432 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/677346c055844c3ca3d887d311729063 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/677346c055844c3ca3d887d311729063 2023-04-28 08:56:08,433 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7ff1a72d3c844a84944bab10a5e769b5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/7ff1a72d3c844a84944bab10a5e769b5 2023-04-28 08:56:08,434 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b6642c4036564eb3b9d89e9b4c65a4be to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b6642c4036564eb3b9d89e9b4c65a4be 2023-04-28 08:56:08,435 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/99761ee611be46f98e4c616dd969a566 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/99761ee611be46f98e4c616dd969a566 2023-04-28 08:56:08,436 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0b81167743fb4a06a7d4102641232133 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0b81167743fb4a06a7d4102641232133 2023-04-28 08:56:08,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4be67934964c4753a44960b31ee4b366 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4be67934964c4753a44960b31ee4b366 2023-04-28 08:56:08,437 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0d7d9b7bd8304542ab9570ca23985523 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/0d7d9b7bd8304542ab9570ca23985523 2023-04-28 08:56:08,438 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ff2f48831e724ccd8321c0575754e748 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ff2f48831e724ccd8321c0575754e748 2023-04-28 08:56:08,439 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a3c0132a98e1434db6fff1693e313526 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a3c0132a98e1434db6fff1693e313526 2023-04-28 08:56:08,440 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/de9ed13639d0497fa5875f3757b2f75d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/de9ed13639d0497fa5875f3757b2f75d 2023-04-28 08:56:08,441 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bc9f3a91cfcc45749b8b7accd5c8d465 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bc9f3a91cfcc45749b8b7accd5c8d465 2023-04-28 08:56:08,442 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b84d2662a425430281e4ce3518f9e8d3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b84d2662a425430281e4ce3518f9e8d3 2023-04-28 08:56:08,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b9daab6096314f1aa515220dd76bbbd6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/b9daab6096314f1aa515220dd76bbbd6 2023-04-28 08:56:08,443 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8e69047faab24b2aa953e1899b7fb9c4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8e69047faab24b2aa953e1899b7fb9c4 2023-04-28 08:56:08,444 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/6a6bcaea52734bc9a98dc53fa1ec652c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/6a6bcaea52734bc9a98dc53fa1ec652c 2023-04-28 08:56:08,445 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1d4aeb4962a848b0971d83b150298704 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1d4aeb4962a848b0971d83b150298704 2023-04-28 08:56:08,446 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/c0d348127c1e49a4bbf4270bc5d438d0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/c0d348127c1e49a4bbf4270bc5d438d0 2023-04-28 08:56:08,447 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ccc984ede72c4c46ad1b9c22056db54d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ccc984ede72c4c46ad1b9c22056db54d 2023-04-28 08:56:08,448 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/80718ec2d611462a93dd734f48c9d115 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/80718ec2d611462a93dd734f48c9d115 2023-04-28 08:56:08,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4ecb115d975b466da38bff29f6836dd9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4ecb115d975b466da38bff29f6836dd9 2023-04-28 08:56:08,449 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bcf49bec3a88415dad13f75c5083ced4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bcf49bec3a88415dad13f75c5083ced4 2023-04-28 08:56:08,450 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/368f1a33b0644515bd83bc68015b54f8 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/368f1a33b0644515bd83bc68015b54f8 2023-04-28 08:56:08,452 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4eea18df87a14394b4a68a4ec547c04e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/4eea18df87a14394b4a68a4ec547c04e 2023-04-28 08:56:08,453 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/f5e38095f13047018f64ae4627a35e5b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/f5e38095f13047018f64ae4627a35e5b 2023-04-28 08:56:08,454 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b97da1484144a0a9121bf7d0a4aa4fd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/5b97da1484144a0a9121bf7d0a4aa4fd 2023-04-28 08:56:08,455 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/d6fa05c1bd3f469dbf9a3942db4da884 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/d6fa05c1bd3f469dbf9a3942db4da884 2023-04-28 08:56:08,455 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ae5d27f7ee5b476b8c84eec37c7c4483 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/ae5d27f7ee5b476b8c84eec37c7c4483 2023-04-28 08:56:08,456 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/539979cec34f45a79adf4b5c2e739c25 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/539979cec34f45a79adf4b5c2e739c25 2023-04-28 08:56:08,457 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/75b6db7ae95440fbbb1fef47ab3340a3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/75b6db7ae95440fbbb1fef47ab3340a3 2023-04-28 08:56:08,458 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a258b8a88a2642d1b5554b0fe3f5ebd9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/a258b8a88a2642d1b5554b0fe3f5ebd9 2023-04-28 08:56:08,459 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/3555b8d1ae254730b7e900a18f23ffee to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/3555b8d1ae254730b7e900a18f23ffee 2023-04-28 08:56:08,460 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/be97f72aedd843be98fcdc33470f36ba to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/be97f72aedd843be98fcdc33470f36ba 2023-04-28 08:56:08,460 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/9ad8b91fd24e4d68b99e5b82882892d5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/9ad8b91fd24e4d68b99e5b82882892d5 2023-04-28 08:56:08,461 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/55e3ba2805cb4ad9b07584f55d71ff5b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/55e3ba2805cb4ad9b07584f55d71ff5b 2023-04-28 08:56:08,462 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/18e3b4374ead448ebea4c15234231ad5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/18e3b4374ead448ebea4c15234231ad5 2023-04-28 08:56:08,463 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8ad04673b7cd44459bbac132b87995f3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/8ad04673b7cd44459bbac132b87995f3 2023-04-28 08:56:08,464 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/da6e2c2057cb43f48952285bf43c7b68 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/da6e2c2057cb43f48952285bf43c7b68 2023-04-28 08:56:08,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bbec9ab4814048ff866775b308c57549 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/bbec9ab4814048ff866775b308c57549 2023-04-28 08:56:08,465 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/e969bf63383941e6a817c62d16f9ed6b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/e969bf63383941e6a817c62d16f9ed6b 2023-04-28 08:56:08,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9d666327a74b41b4bd4d455733e39815, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/abd5cdac1698419a91159d44aec5768f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/60b242f5a0e04d2fad9e60ad4a23a0e7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1c7b9ddade604942aba4c8e1b1b44e7b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/8c18782ea7d4409e8a5ece1ea3d72a03, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1660d23ef1514d9dbbc6d3786b01ded7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/c7beb9f4a1204948926b5c9f4bc690b0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4ac7aee9403746ba952b55372b872d45, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/ac340fd6f6a04cc499cc5e246b03998b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1de1aed1bfab4fe2b45cee56f5b7269b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9221fb359508489bb9aabc8e828301d1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6a60090e453248cf8d87cbf5ae5fc8f0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/394ec33ad2af431f8466dcb0d160651b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/534c4446c1d3483d8a03c81833f61749, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/540479f61dc145bdbd7b9d2b8722fc33, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/42edfa104a5a4d65a327b15328c454a2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4faded1590364b7989610d840b76824c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bb26374e375144c5b34649c9be3e948b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/d82048320b284da28ad88a280bbe39c3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/90385840a55f4d22b343b48f31eb4909, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/38cd3cd978a24618b6537eebac3335d5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9ca8f769f430499a88295bd324c6a0fc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dd67e3bbc35b42cdaf50f50936875e01, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bda1395ccaa14bdab7ffedb9c1de3af9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7b7a6dcac3224388b05582c2e563192e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/73fcae09d3c0432b897f210dd119aef3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/3e95e1d9b214418ab375c638f3b08c23, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/267253bed5114175b4a15da843cef773, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/86828a49546c42568211127b53910fd4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/de90d8afb6e54df0be21746debde9f67, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/01e4640d8a114dd1a81e75aea88c735b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dcc6db6ea60b4f249c6e311ebbc572b1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a6fc4f1f5b0a47de8d6cda764a8ed690, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9503bc92acd74d019e997729f60ef89e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7c8f9ffde88241d58be70e39159ed7a9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/e396e490063a4bba9fe894074c9114af, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a5699f72f5b2408bbe577b27abb94862, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/15cff6bf2ec749ce8c9fdfef67d1824a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/2b72b2048c7546899a7e1ea091fb7466, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/76c9b1f905a044b7ac2e9eb72e1d1100, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6b46e63331714cecbdd76d08c5f3a76a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/513e07579dcf47a4bf2fad48d800cc11, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/28eedf282f1644db957abbdec44b1750, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/339d86d71303418f996dae348f0cb8f4] to archive 2023-04-28 08:56:08,469 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:56:08,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9d666327a74b41b4bd4d455733e39815 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9d666327a74b41b4bd4d455733e39815 2023-04-28 08:56:08,471 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/abd5cdac1698419a91159d44aec5768f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/abd5cdac1698419a91159d44aec5768f 2023-04-28 08:56:08,472 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/60b242f5a0e04d2fad9e60ad4a23a0e7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/60b242f5a0e04d2fad9e60ad4a23a0e7 2023-04-28 08:56:08,473 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1c7b9ddade604942aba4c8e1b1b44e7b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1c7b9ddade604942aba4c8e1b1b44e7b 2023-04-28 08:56:08,474 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/8c18782ea7d4409e8a5ece1ea3d72a03 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/8c18782ea7d4409e8a5ece1ea3d72a03 2023-04-28 08:56:08,475 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1660d23ef1514d9dbbc6d3786b01ded7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1660d23ef1514d9dbbc6d3786b01ded7 2023-04-28 08:56:08,476 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/c7beb9f4a1204948926b5c9f4bc690b0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/c7beb9f4a1204948926b5c9f4bc690b0 2023-04-28 08:56:08,477 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4ac7aee9403746ba952b55372b872d45 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4ac7aee9403746ba952b55372b872d45 2023-04-28 08:56:08,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/ac340fd6f6a04cc499cc5e246b03998b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/ac340fd6f6a04cc499cc5e246b03998b 2023-04-28 08:56:08,478 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1de1aed1bfab4fe2b45cee56f5b7269b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/1de1aed1bfab4fe2b45cee56f5b7269b 2023-04-28 08:56:08,479 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9221fb359508489bb9aabc8e828301d1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9221fb359508489bb9aabc8e828301d1 2023-04-28 08:56:08,480 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6a60090e453248cf8d87cbf5ae5fc8f0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6a60090e453248cf8d87cbf5ae5fc8f0 2023-04-28 08:56:08,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/394ec33ad2af431f8466dcb0d160651b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/394ec33ad2af431f8466dcb0d160651b 2023-04-28 08:56:08,481 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/534c4446c1d3483d8a03c81833f61749 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/534c4446c1d3483d8a03c81833f61749 2023-04-28 08:56:08,482 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/540479f61dc145bdbd7b9d2b8722fc33 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/540479f61dc145bdbd7b9d2b8722fc33 2023-04-28 08:56:08,483 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/42edfa104a5a4d65a327b15328c454a2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/42edfa104a5a4d65a327b15328c454a2 2023-04-28 08:56:08,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4faded1590364b7989610d840b76824c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/4faded1590364b7989610d840b76824c 2023-04-28 08:56:08,484 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bb26374e375144c5b34649c9be3e948b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bb26374e375144c5b34649c9be3e948b 2023-04-28 08:56:08,485 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/d82048320b284da28ad88a280bbe39c3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/d82048320b284da28ad88a280bbe39c3 2023-04-28 08:56:08,486 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/90385840a55f4d22b343b48f31eb4909 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/90385840a55f4d22b343b48f31eb4909 2023-04-28 08:56:08,487 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/38cd3cd978a24618b6537eebac3335d5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/38cd3cd978a24618b6537eebac3335d5 2023-04-28 08:56:08,488 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9ca8f769f430499a88295bd324c6a0fc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9ca8f769f430499a88295bd324c6a0fc 2023-04-28 08:56:08,489 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dd67e3bbc35b42cdaf50f50936875e01 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dd67e3bbc35b42cdaf50f50936875e01 2023-04-28 08:56:08,490 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bda1395ccaa14bdab7ffedb9c1de3af9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/bda1395ccaa14bdab7ffedb9c1de3af9 2023-04-28 08:56:08,491 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7b7a6dcac3224388b05582c2e563192e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7b7a6dcac3224388b05582c2e563192e 2023-04-28 08:56:08,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/73fcae09d3c0432b897f210dd119aef3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/73fcae09d3c0432b897f210dd119aef3 2023-04-28 08:56:08,492 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/3e95e1d9b214418ab375c638f3b08c23 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/3e95e1d9b214418ab375c638f3b08c23 2023-04-28 08:56:08,493 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/267253bed5114175b4a15da843cef773 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/267253bed5114175b4a15da843cef773 2023-04-28 08:56:08,494 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/86828a49546c42568211127b53910fd4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/86828a49546c42568211127b53910fd4 2023-04-28 08:56:08,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/de90d8afb6e54df0be21746debde9f67 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/de90d8afb6e54df0be21746debde9f67 2023-04-28 08:56:08,495 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/01e4640d8a114dd1a81e75aea88c735b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/01e4640d8a114dd1a81e75aea88c735b 2023-04-28 08:56:08,496 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dcc6db6ea60b4f249c6e311ebbc572b1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/dcc6db6ea60b4f249c6e311ebbc572b1 2023-04-28 08:56:08,497 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a6fc4f1f5b0a47de8d6cda764a8ed690 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a6fc4f1f5b0a47de8d6cda764a8ed690 2023-04-28 08:56:08,498 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9503bc92acd74d019e997729f60ef89e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/9503bc92acd74d019e997729f60ef89e 2023-04-28 08:56:08,499 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7c8f9ffde88241d58be70e39159ed7a9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/7c8f9ffde88241d58be70e39159ed7a9 2023-04-28 08:56:08,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/e396e490063a4bba9fe894074c9114af to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/e396e490063a4bba9fe894074c9114af 2023-04-28 08:56:08,500 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a5699f72f5b2408bbe577b27abb94862 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a5699f72f5b2408bbe577b27abb94862 2023-04-28 08:56:08,501 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/15cff6bf2ec749ce8c9fdfef67d1824a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/15cff6bf2ec749ce8c9fdfef67d1824a 2023-04-28 08:56:08,502 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/2b72b2048c7546899a7e1ea091fb7466 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/2b72b2048c7546899a7e1ea091fb7466 2023-04-28 08:56:08,503 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/76c9b1f905a044b7ac2e9eb72e1d1100 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/76c9b1f905a044b7ac2e9eb72e1d1100 2023-04-28 08:56:08,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6b46e63331714cecbdd76d08c5f3a76a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/6b46e63331714cecbdd76d08c5f3a76a 2023-04-28 08:56:08,504 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/513e07579dcf47a4bf2fad48d800cc11 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/513e07579dcf47a4bf2fad48d800cc11 2023-04-28 08:56:08,505 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/28eedf282f1644db957abbdec44b1750 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/28eedf282f1644db957abbdec44b1750 2023-04-28 08:56:08,506 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/339d86d71303418f996dae348f0cb8f4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/339d86d71303418f996dae348f0cb8f4 2023-04-28 08:56:08,509 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7789ef02b0754b1fbd5673470827e9f9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/76dbb612b7c345b08a86bde31c1d66a4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e2cccc3955774b8bb2b868fd48b27a23, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f4bc9669519645868e45d0c1bc6b48f8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/bdfefb0571d740cab050cf0961e252d8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/19c79da6f07347028537e98cb5508cc4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/432c7f5c7ded4b39ae77a0c864e2e5d9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c2f5115933564b14ab9428686563bf61, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/576b1c0a232b4fcfb5cf728e50cd8fa2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/2dbeb15e52a044fdb78daec071ec19fd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/fd8b3782f28646f9913442d42938c4ae, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/3d313fb7389846d7a6d21d6f8dd41d2f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/018647ea27cd43cca158fb97ddba535d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/732a1bfcdea340298fde326b7b063acf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e7393f532a564cd1afce2b99dde3fa9c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/91b866f2d3fe40eb8e0e220b2426590a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/793c574135ea4997a61940dcd1910c6c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d0c922f4ed4447b884f14f862836dde2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8b9db8f3a23e46cebe22ed1811a8e68c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7ef28b0166404553b6e0718f06fd9337, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d4732aa2b5ac4e718ce01accf0dbcf05, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/87fb372e3ee249a58586c56cd286549b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/0ea754ad5efc40129ea71e523174c2cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b0448d006800485985e39f616094b9ba, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/271b3285625f4d4f9e2ee1a82b212002, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d98de485917844e1b9c6f23929e84db9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b174c8da109a45098ab8e4ad7f509b20, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4d13ad7936a841498351c14e299b5f32, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/897f873a1849402a876a089c1d5ddb4c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d5c0179591444833bb5515cc840a6301, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/df2fc9c5bce54ac085bc6a158190916c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a68935bd7fb34409b87f7bc65ca71e9e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f10ce73087e244578a3a2f3815e99d70, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4a3633bf737c4bdaacaa661b11365b37, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c57bc66aa1194926a7b9500cc98d6a6f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/badba45e382240879ea14fa363a81b47, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a2ab5cd0fa4d4bb091afb511fe24b82f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7d776089e6c140f5abbe453311479938, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/9051f12165e148799ce855f79e7dd5ac, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/5a0e747f5e3641a6952797520af219ae, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/56b5c6eac46045ccaa5ad71c1447a5ea, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/44f57869087048e6ba1e1023241e3218] to archive 2023-04-28 08:56:08,510 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:56:08,511 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7789ef02b0754b1fbd5673470827e9f9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7789ef02b0754b1fbd5673470827e9f9 2023-04-28 08:56:08,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/76dbb612b7c345b08a86bde31c1d66a4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/76dbb612b7c345b08a86bde31c1d66a4 2023-04-28 08:56:08,512 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e2cccc3955774b8bb2b868fd48b27a23 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e2cccc3955774b8bb2b868fd48b27a23 2023-04-28 08:56:08,513 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f4bc9669519645868e45d0c1bc6b48f8 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f4bc9669519645868e45d0c1bc6b48f8 2023-04-28 08:56:08,514 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/bdfefb0571d740cab050cf0961e252d8 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/bdfefb0571d740cab050cf0961e252d8 2023-04-28 08:56:08,515 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/19c79da6f07347028537e98cb5508cc4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/19c79da6f07347028537e98cb5508cc4 2023-04-28 08:56:08,516 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/432c7f5c7ded4b39ae77a0c864e2e5d9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/432c7f5c7ded4b39ae77a0c864e2e5d9 2023-04-28 08:56:08,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c2f5115933564b14ab9428686563bf61 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c2f5115933564b14ab9428686563bf61 2023-04-28 08:56:08,517 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/576b1c0a232b4fcfb5cf728e50cd8fa2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/576b1c0a232b4fcfb5cf728e50cd8fa2 2023-04-28 08:56:08,518 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/2dbeb15e52a044fdb78daec071ec19fd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/2dbeb15e52a044fdb78daec071ec19fd 2023-04-28 08:56:08,519 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/fd8b3782f28646f9913442d42938c4ae to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/fd8b3782f28646f9913442d42938c4ae 2023-04-28 08:56:08,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/3d313fb7389846d7a6d21d6f8dd41d2f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/3d313fb7389846d7a6d21d6f8dd41d2f 2023-04-28 08:56:08,520 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/018647ea27cd43cca158fb97ddba535d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/018647ea27cd43cca158fb97ddba535d 2023-04-28 08:56:08,521 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/732a1bfcdea340298fde326b7b063acf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/732a1bfcdea340298fde326b7b063acf 2023-04-28 08:56:08,522 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e7393f532a564cd1afce2b99dde3fa9c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/e7393f532a564cd1afce2b99dde3fa9c 2023-04-28 08:56:08,523 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/91b866f2d3fe40eb8e0e220b2426590a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/91b866f2d3fe40eb8e0e220b2426590a 2023-04-28 08:56:08,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/793c574135ea4997a61940dcd1910c6c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/793c574135ea4997a61940dcd1910c6c 2023-04-28 08:56:08,524 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d0c922f4ed4447b884f14f862836dde2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d0c922f4ed4447b884f14f862836dde2 2023-04-28 08:56:08,525 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8b9db8f3a23e46cebe22ed1811a8e68c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8b9db8f3a23e46cebe22ed1811a8e68c 2023-04-28 08:56:08,526 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7ef28b0166404553b6e0718f06fd9337 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7ef28b0166404553b6e0718f06fd9337 2023-04-28 08:56:08,527 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d4732aa2b5ac4e718ce01accf0dbcf05 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d4732aa2b5ac4e718ce01accf0dbcf05 2023-04-28 08:56:08,528 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/87fb372e3ee249a58586c56cd286549b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/87fb372e3ee249a58586c56cd286549b 2023-04-28 08:56:08,529 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/0ea754ad5efc40129ea71e523174c2cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/0ea754ad5efc40129ea71e523174c2cb 2023-04-28 08:56:08,530 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b0448d006800485985e39f616094b9ba to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b0448d006800485985e39f616094b9ba 2023-04-28 08:56:08,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/271b3285625f4d4f9e2ee1a82b212002 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/271b3285625f4d4f9e2ee1a82b212002 2023-04-28 08:56:08,531 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d98de485917844e1b9c6f23929e84db9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d98de485917844e1b9c6f23929e84db9 2023-04-28 08:56:08,532 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b174c8da109a45098ab8e4ad7f509b20 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/b174c8da109a45098ab8e4ad7f509b20 2023-04-28 08:56:08,533 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4d13ad7936a841498351c14e299b5f32 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4d13ad7936a841498351c14e299b5f32 2023-04-28 08:56:08,534 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/897f873a1849402a876a089c1d5ddb4c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/897f873a1849402a876a089c1d5ddb4c 2023-04-28 08:56:08,535 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d5c0179591444833bb5515cc840a6301 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/d5c0179591444833bb5515cc840a6301 2023-04-28 08:56:08,536 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/df2fc9c5bce54ac085bc6a158190916c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/df2fc9c5bce54ac085bc6a158190916c 2023-04-28 08:56:08,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a68935bd7fb34409b87f7bc65ca71e9e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a68935bd7fb34409b87f7bc65ca71e9e 2023-04-28 08:56:08,537 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f10ce73087e244578a3a2f3815e99d70 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/f10ce73087e244578a3a2f3815e99d70 2023-04-28 08:56:08,538 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4a3633bf737c4bdaacaa661b11365b37 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4a3633bf737c4bdaacaa661b11365b37 2023-04-28 08:56:08,539 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c57bc66aa1194926a7b9500cc98d6a6f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/c57bc66aa1194926a7b9500cc98d6a6f 2023-04-28 08:56:08,540 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/badba45e382240879ea14fa363a81b47 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/badba45e382240879ea14fa363a81b47 2023-04-28 08:56:08,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a2ab5cd0fa4d4bb091afb511fe24b82f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/a2ab5cd0fa4d4bb091afb511fe24b82f 2023-04-28 08:56:08,541 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7d776089e6c140f5abbe453311479938 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/7d776089e6c140f5abbe453311479938 2023-04-28 08:56:08,542 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/9051f12165e148799ce855f79e7dd5ac to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/9051f12165e148799ce855f79e7dd5ac 2023-04-28 08:56:08,543 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/5a0e747f5e3641a6952797520af219ae to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/5a0e747f5e3641a6952797520af219ae 2023-04-28 08:56:08,544 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/56b5c6eac46045ccaa5ad71c1447a5ea to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/56b5c6eac46045ccaa5ad71c1447a5ea 2023-04-28 08:56:08,545 DEBUG [StoreCloser-TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/44f57869087048e6ba1e1023241e3218 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/44f57869087048e6ba1e1023241e3218 2023-04-28 08:56:08,548 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/recovered.edits/829.seqid, newMaxSeqId=829, maxSeqId=1 2023-04-28 08:56:08,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4. 2023-04-28 08:56:08,550 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 2ff295b65a6980314fe434234aeef4a4: 2023-04-28 08:56:08,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:56:08,551 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2ff295b65a6980314fe434234aeef4a4, regionState=CLOSED 2023-04-28 08:56:08,552 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672168551"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672168551"}]},"ts":"1682672168551"} 2023-04-28 08:56:08,554 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-28 08:56:08,554 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 2ff295b65a6980314fe434234aeef4a4, server=jenkins-hbase6.apache.org,40909,1682672018517 in 1.5490 sec 2023-04-28 08:56:08,555 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-28 08:56:08,555 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ff295b65a6980314fe434234aeef4a4, UNASSIGN in 1.5530 sec 2023-04-28 08:56:08,555 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672168555"}]},"ts":"1682672168555"} 2023-04-28 08:56:08,556 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 08:56:08,558 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 08:56:08,559 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6600 sec 2023-04-28 08:56:09,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-28 08:56:09,007 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-28 08:56:09,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete TestAcidGuarantees 2023-04-28 08:56:09,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:09,010 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:09,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-28 08:56:09,011 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:09,014 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:56:09,016 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/recovered.edits] 2023-04-28 08:56:09,018 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1286f830fd8843db90cd7a4e93246909 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/1286f830fd8843db90cd7a4e93246909 2023-04-28 08:56:09,019 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/498a1fe142a84f888ea8d2b01ebc7f97 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/A/498a1fe142a84f888ea8d2b01ebc7f97 2023-04-28 08:56:09,021 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/734f7d0fba274d4cb4af2534598c18ea to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/734f7d0fba274d4cb4af2534598c18ea 2023-04-28 08:56:09,022 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a8c4ddb8daf04b48b74c9745eb10b16f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/B/a8c4ddb8daf04b48b74c9745eb10b16f 2023-04-28 08:56:09,024 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4bcbc00454bd4b8cbaf8ad4ad03b7adf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/4bcbc00454bd4b8cbaf8ad4ad03b7adf 2023-04-28 08:56:09,025 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8d3b9df37a4a4eacbceb46550a373650 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/8d3b9df37a4a4eacbceb46550a373650 2023-04-28 08:56:09,025 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/930249343af140f8a68800b285595620 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/C/930249343af140f8a68800b285595620 2023-04-28 08:56:09,028 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/recovered.edits/829.seqid to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4/recovered.edits/829.seqid 2023-04-28 08:56:09,028 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/2ff295b65a6980314fe434234aeef4a4 2023-04-28 08:56:09,028 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:56:09,030 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:09,031 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 08:56:09,032 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 08:56:09,033 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:09,033 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 08:56:09,033 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682672169033"}]},"ts":"9223372036854775807"} 2023-04-28 08:56:09,036 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 08:56:09,036 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2ff295b65a6980314fe434234aeef4a4, NAME => 'TestAcidGuarantees,,1682672137277.2ff295b65a6980314fe434234aeef4a4.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 08:56:09,036 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 08:56:09,036 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682672169036"}]},"ts":"9223372036854775807"} 2023-04-28 08:56:09,037 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 08:56:09,038 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:09,039 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 30 msec 2023-04-28 08:56:09,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-28 08:56:09,112 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-28 08:56:09,124 INFO [Listener at localhost/39717] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 224) - Thread LEAK? -, OpenFileDescriptor=531 (was 518) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=426 (was 360) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=6426 (was 6026) - AvailableMemoryMB LEAK? - 2023-04-28 08:56:09,136 INFO [Listener at localhost/39717] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=426, ProcessCount=170, AvailableMemoryMB=6425 2023-04-28 08:56:09,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:56:09,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$4(2112): Client=jenkins//172.31.2.20 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:56:09,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:09,140 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-28 08:56:09,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(697): Client=jenkins//172.31.2.20 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-28 08:56:09,141 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-28 08:56:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 08:56:09,142 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:09,142 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb empty. 2023-04-28 08:56:09,143 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:09,143 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:56:09,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 08:56:09,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 08:56:09,549 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:56:09,551 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 92c5373160b2e33e00d0bc96891cd0cb, NAME => 'TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp 2023-04-28 08:56:09,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 08:56:09,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:56:09,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 92c5373160b2e33e00d0bc96891cd0cb, disabling compactions & flushes 2023-04-28 08:56:09,956 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:09,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:09,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. after waiting 0 ms 2023-04-28 08:56:09,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:09,956 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:09,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:09,958 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-28 08:56:09,959 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672169959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672169959"}]},"ts":"1682672169959"} 2023-04-28 08:56:09,960 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-28 08:56:09,960 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-28 08:56:09,961 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672169961"}]},"ts":"1682672169961"} 2023-04-28 08:56:09,961 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-28 08:56:09,964 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, ASSIGN}] 2023-04-28 08:56:09,965 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, ASSIGN 2023-04-28 08:56:09,966 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, ASSIGN; state=OFFLINE, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=false 2023-04-28 08:56:10,117 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:10,117 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672170117"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672170117"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672170117"}]},"ts":"1682672170117"} 2023-04-28 08:56:10,118 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:56:10,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 08:56:10,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:10,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 92c5373160b2e33e00d0bc96891cd0cb, NAME => 'TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:56:10,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:56:10,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,276 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,277 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A 2023-04-28 08:56:10,277 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A 2023-04-28 08:56:10,278 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:56:10,278 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92c5373160b2e33e00d0bc96891cd0cb columnFamilyName A 2023-04-28 08:56:10,279 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(310): Store=92c5373160b2e33e00d0bc96891cd0cb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:56:10,279 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,280 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B 2023-04-28 08:56:10,280 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B 2023-04-28 08:56:10,280 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:56:10,280 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92c5373160b2e33e00d0bc96891cd0cb columnFamilyName B 2023-04-28 08:56:10,281 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(310): Store=92c5373160b2e33e00d0bc96891cd0cb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:56:10,281 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,282 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C 2023-04-28 08:56:10,282 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C 2023-04-28 08:56:10,282 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:56:10,282 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92c5373160b2e33e00d0bc96891cd0cb columnFamilyName C 2023-04-28 08:56:10,282 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(310): Store=92c5373160b2e33e00d0bc96891cd0cb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:56:10,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:10,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:56:10,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:10,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-28 08:56:10,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 92c5373160b2e33e00d0bc96891cd0cb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59571285, jitterRate=-0.11231867969036102}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:56:10,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:10,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., pid=56, masterSystemTime=1682672170270 2023-04-28 08:56:10,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:10,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:10,290 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:10,290 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672170290"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672170290"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672170290"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672170290"}]},"ts":"1682672170290"} 2023-04-28 08:56:10,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-28 08:56:10,292 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 in 173 msec 2023-04-28 08:56:10,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-28 08:56:10,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, ASSIGN in 328 msec 2023-04-28 08:56:10,294 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-28 08:56:10,294 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672170294"}]},"ts":"1682672170294"} 2023-04-28 08:56:10,295 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-28 08:56:10,297 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-28 08:56:10,298 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-28 08:56:11,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-28 08:56:11,246 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-28 08:56:11,247 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x470742aa to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:11,252 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ced4ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:11,253 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:11,254 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:11,255 DEBUG [Listener at localhost/39717] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-28 08:56:11,256 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:37550, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-28 08:56:11,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-28 08:56:11,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$12(2454): Client=jenkins//172.31.2.20 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-28 08:56:11,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:11,671 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 08:56:11,672 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-28 08:56:11,672 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-28 08:56:11,674 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-28 08:56:11,676 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, REOPEN/MOVE}] 2023-04-28 08:56:11,676 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, REOPEN/MOVE 2023-04-28 08:56:11,677 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:11,677 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672171677"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672171677"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672171677"}]},"ts":"1682672171677"} 2023-04-28 08:56:11,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:56:11,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:11,830 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 92c5373160b2e33e00d0bc96891cd0cb, disabling compactions & flushes 2023-04-28 08:56:11,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:11,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:11,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. after waiting 0 ms 2023-04-28 08:56:11,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:11,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-28 08:56:11,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:11,834 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:11,834 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 92c5373160b2e33e00d0bc96891cd0cb to self. 2023-04-28 08:56:11,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:11,836 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=CLOSED 2023-04-28 08:56:11,836 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672171836"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672171836"}]},"ts":"1682672171836"} 2023-04-28 08:56:11,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-28 08:56:11,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 in 159 msec 2023-04-28 08:56:11,838 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase6.apache.org,40909,1682672018517; forceNewPlan=false, retain=true 2023-04-28 08:56:11,989 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=OPENING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:11,989 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672171989"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672171989"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672171989"}]},"ts":"1682672171989"} 2023-04-28 08:56:11,990 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:56:12,144 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:12,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 92c5373160b2e33e00d0bc96891cd0cb, NAME => 'TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.', STARTKEY => '', ENDKEY => ''} 2023-04-28 08:56:12,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-28 08:56:12,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7894): checking encryption for 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(7897): checking classloading for 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,146 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,147 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A 2023-04-28 08:56:12,147 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A 2023-04-28 08:56:12,147 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:56:12,147 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92c5373160b2e33e00d0bc96891cd0cb columnFamilyName A 2023-04-28 08:56:12,148 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(310): Store=92c5373160b2e33e00d0bc96891cd0cb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:56:12,148 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,149 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B 2023-04-28 08:56:12,149 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B 2023-04-28 08:56:12,149 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:56:12,149 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92c5373160b2e33e00d0bc96891cd0cb columnFamilyName B 2023-04-28 08:56:12,150 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(310): Store=92c5373160b2e33e00d0bc96891cd0cb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:56:12,150 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,150 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C 2023-04-28 08:56:12,150 DEBUG [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C 2023-04-28 08:56:12,151 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-28 08:56:12,151 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92c5373160b2e33e00d0bc96891cd0cb columnFamilyName C 2023-04-28 08:56:12,151 INFO [StoreOpener-92c5373160b2e33e00d0bc96891cd0cb-1] regionserver.HStore(310): Store=92c5373160b2e33e00d0bc96891cd0cb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-28 08:56:12,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:12,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-28 08:56:12,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1055): writing seq id for 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1072): Opened 92c5373160b2e33e00d0bc96891cd0cb; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69457978, jitterRate=0.03500452637672424}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-28 08:56:12,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(965): Region open journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:12,157 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., pid=61, masterSystemTime=1682672172142 2023-04-28 08:56:12,158 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:12,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase6:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:12,158 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,159 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672172158"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682672172158"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682672172158"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682672172158"}]},"ts":"1682672172158"} 2023-04-28 08:56:12,160 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-28 08:56:12,160 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 in 169 msec 2023-04-28 08:56:12,161 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-28 08:56:12,162 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, REOPEN/MOVE in 484 msec 2023-04-28 08:56:12,163 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-28 08:56:12,163 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-28 08:56:12,165 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 906 msec 2023-04-28 08:56:12,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-28 08:56:12,166 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3503b4 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,171 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63da10ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,171 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x6f06dd50 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,175 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59185ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,175 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e8654 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,179 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62b3caa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,179 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x4db22237 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,184 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b5b1f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,184 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x08b57aae to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,188 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@544a4215, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,188 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,193 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,193 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,201 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,201 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x3b59bdcf to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,208 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a470a3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,209 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x3d765193 to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,212 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bb72437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,212 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(139): Connect 0x32bf16cd to 127.0.0.1:54655 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-28 08:56:12,218 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dca23c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-28 08:56:12,229 DEBUG [hconnection-0x6dd3700d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,229 DEBUG [hconnection-0x2b78a671-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,229 DEBUG [hconnection-0x5920204f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,230 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,231 DEBUG [hconnection-0x76384951-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,232 DEBUG [hconnection-0x37523742-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,232 DEBUG [hconnection-0x23d90ef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,232 DEBUG [hconnection-0x218b55b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,235 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59116, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,233 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,233 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,232 DEBUG [hconnection-0x2cc0fd84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,237 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,238 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,244 DEBUG [hconnection-0x7bcfe19c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,245 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,244 DEBUG [hconnection-0x597cc4d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-28 08:56:12,245 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59132, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:12,249 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59130, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,249 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.20:59144, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-28 08:56:12,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:12,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:12,255 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:12,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:12,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:12,255 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:12,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,256 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,257 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:12,257 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:12,258 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,258 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:12,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:12,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:12,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:12,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:12,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:12,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:12,259 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:12,259 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:12,259 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:12,260 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:56:12,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:12,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:12,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:12,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:12,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:12,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:12,261 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:12,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672232276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672172266, max timestamp=1682672172277Num uniques 859; 2023-04-28 08:56:12,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:56:12,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672232276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1682672172266, max timestamp=1682672172277Num uniques 859; 2023-04-28 08:56:12,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-28 08:56:12,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1682672172266, max timestamp=1682672172277Num uniques 859; 2023-04-28 08:56:12,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-28 08:56:12,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672232293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,296 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672232293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672232295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,298 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042815d4febd2f324d11bcc5e121903c826d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042815d4febd2f324d11bcc5e121903c826d 2023-04-28 08:56:12,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672232379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672232380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672232397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672232397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672232398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672232583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672232584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672232598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672232600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672232601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,716 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/e9562ce0b8644a8b9092181f1788cc57 2023-04-28 08:56:12,726 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/eee7e57bc5564e318300c22e18e63594 2023-04-28 08:56:12,736 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a07fc6e7427d4db88791253fac3d8cff 2023-04-28 08:56:12,740 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/e9562ce0b8644a8b9092181f1788cc57 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e9562ce0b8644a8b9092181f1788cc57 2023-04-28 08:56:12,745 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e9562ce0b8644a8b9092181f1788cc57, entries=200, sequenceid=15, filesize=31.9 K 2023-04-28 08:56:12,746 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/eee7e57bc5564e318300c22e18e63594 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/eee7e57bc5564e318300c22e18e63594 2023-04-28 08:56:12,750 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/eee7e57bc5564e318300c22e18e63594, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 08:56:12,750 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a07fc6e7427d4db88791253fac3d8cff as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a07fc6e7427d4db88791253fac3d8cff 2023-04-28 08:56:12,756 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a07fc6e7427d4db88791253fac3d8cff, entries=150, sequenceid=15, filesize=11.6 K 2023-04-28 08:56:12,756 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.31 KB/132415 for 92c5373160b2e33e00d0bc96891cd0cb in 496ms, sequenceid=15, compaction requested=false 2023-04-28 08:56:12,756 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-28 08:56:12,757 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:12,757 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:12,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:12,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:12,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:12,757 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:12,760 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,760 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:12,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:12,760 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,760 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:12,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1a43f6[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:12,762 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:12,762 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,763 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:12,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:12,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:12,764 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:12,765 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:12,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:12,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:12,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:12,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:12,768 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:12,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:12,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:12,768 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:12,768 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:12,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:12,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:12,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:12,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:12,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,774 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:12,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:12,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:12,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,777 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:12,777 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:12,777 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:12,777 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:12,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:12,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:12,878 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:12,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:12,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:12,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:12,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:12,881 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:12,881 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:12,881 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:12,881 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:12,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,883 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:12,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:12,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,883 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:12,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:12,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:12,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:12,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:12,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:12,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:12,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:12,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:12,885 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=129.31 KB heapSize=364.54 KB 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:12,885 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:12,907 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304285a0b63ce13c2423392d95348d7f7254c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304285a0b63ce13c2423392d95348d7f7254c 2023-04-28 08:56:12,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:12,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:12,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672232916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,923 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/a45f472eef0248d7b186b74a028dedb1 2023-04-28 08:56:12,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672232920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672232921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672232921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:12,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672232921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:12,938 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/633955da885a4f15858e7a3fa8afda41 2023-04-28 08:56:13,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672233022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672233025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672233025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672233026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672233026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672233226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672233228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672233228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672233228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672233229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,348 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.86 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a2e3c085cfa5403d8894c47b58bd5bfe 2023-04-28 08:56:13,352 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/a45f472eef0248d7b186b74a028dedb1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a45f472eef0248d7b186b74a028dedb1 2023-04-28 08:56:13,356 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a45f472eef0248d7b186b74a028dedb1, entries=150, sequenceid=50, filesize=25.1 K 2023-04-28 08:56:13,357 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/633955da885a4f15858e7a3fa8afda41 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/633955da885a4f15858e7a3fa8afda41 2023-04-28 08:56:13,361 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/633955da885a4f15858e7a3fa8afda41, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 08:56:13,361 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a2e3c085cfa5403d8894c47b58bd5bfe as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a2e3c085cfa5403d8894c47b58bd5bfe 2023-04-28 08:56:13,365 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a2e3c085cfa5403d8894c47b58bd5bfe, entries=150, sequenceid=50, filesize=11.6 K 2023-04-28 08:56:13,366 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.31 KB/132415, heapSize ~364.49 KB/373240, currentSize=60.38 KB/61830 for 92c5373160b2e33e00d0bc96891cd0cb in 481ms, sequenceid=50, compaction requested=false 2023-04-28 08:56:13,366 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:13,366 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:13,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:13,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:13,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:13,366 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:13,367 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,367 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:13,368 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,368 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:13,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b923765[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:13,370 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:13,370 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,371 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:13,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:13,371 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:13,371 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,371 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:13,373 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:13,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:13,373 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,374 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:13,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:13,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:13,376 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:13,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:13,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:13,377 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:13,377 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:13,378 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,378 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:13,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,378 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:13,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:13,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:13,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:13,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:13,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,383 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:13,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:13,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:13,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:13,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:13,383 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,383 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:13,384 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:13,383 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:13,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:13,484 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:13,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:13,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:13,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:13,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:13,487 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:13,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:13,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:13,487 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:13,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,488 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:13,489 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:13,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:13,489 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:13,490 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,490 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:13,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:13,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:13,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:13,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:13,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:13,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:13,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:13,491 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:13,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:13,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:13,491 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-28 08:56:13,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:13,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:13,491 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:13,492 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:13,492 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:13,492 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:13,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:13,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672233544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672233545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672233546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672233546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672233547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672233648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672233648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672233649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672233651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672233651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672233851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672233853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672233853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672233854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:13,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672233856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:13,904 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428c65c391a100c4bd48d2ce6619948479b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c65c391a100c4bd48d2ce6619948479b 2023-04-28 08:56:13,912 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/1c2f60298b6a41eaba47871ed243eb2d 2023-04-28 08:56:14,087 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-28 08:56:14,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672234155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672234157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672234158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672234159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672234160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,322 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/ee388e0d93d74faa92e2b5e94707e238 2023-04-28 08:56:14,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672234658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672234662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672234663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672234665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:14,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672234665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,734 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/5a521c2f67564c0484e8af6ac6420c60 2023-04-28 08:56:14,737 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/1c2f60298b6a41eaba47871ed243eb2d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c2f60298b6a41eaba47871ed243eb2d 2023-04-28 08:56:14,741 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c2f60298b6a41eaba47871ed243eb2d, entries=150, sequenceid=62, filesize=25.1 K 2023-04-28 08:56:14,742 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/ee388e0d93d74faa92e2b5e94707e238 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ee388e0d93d74faa92e2b5e94707e238 2023-04-28 08:56:14,745 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ee388e0d93d74faa92e2b5e94707e238, entries=150, sequenceid=62, filesize=11.6 K 2023-04-28 08:56:14,746 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/5a521c2f67564c0484e8af6ac6420c60 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/5a521c2f67564c0484e8af6ac6420c60 2023-04-28 08:56:14,749 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/5a521c2f67564c0484e8af6ac6420c60, entries=150, sequenceid=62, filesize=11.6 K 2023-04-28 08:56:14,750 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 92c5373160b2e33e00d0bc96891cd0cb in 1259ms, sequenceid=62, compaction requested=true 2023-04-28 08:56:14,750 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:14,750 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:14,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:14,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:14,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:14,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:14,755 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:14,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,755 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:14,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:14,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:14,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3493954c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:14,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:14,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,758 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:14,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:14,758 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:14,758 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,759 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:14,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,759 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:14,760 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:14,760 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:14,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:14,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:14,762 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:14,762 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:14,762 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:14,762 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:14,762 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:14,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,764 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:14,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:14,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,766 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:14,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:14,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:14,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:14,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:14,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:14,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:14,770 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:14,770 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:14,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:14,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:14,770 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,870 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:14,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:14,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:14,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:14,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:14,874 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:14,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:14,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:14,874 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:14,875 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,875 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:14,876 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:14,876 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:14,876 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:14,876 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:14,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:14,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:14,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:14,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:14,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:14,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:14,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:14,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:14,878 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:14,878 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:14,878 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:14,878 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:56:14,878 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:14,879 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:14,879 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:14,879 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:14,879 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:14,879 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:14,896 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428957f27849f2945df83f7439fb058b15a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428957f27849f2945df83f7439fb058b15a 2023-04-28 08:56:14,904 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=82, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/58c6b3edec05463abfef07bf5584641f 2023-04-28 08:56:15,326 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/3c657b110dd24fc6b31e637110a9edff 2023-04-28 08:56:15,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:15,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:15,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672235675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672235675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672235676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672235677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672235678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,736 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/55f8d33effbb43b5b3f312f48f88ee67 2023-04-28 08:56:15,742 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/58c6b3edec05463abfef07bf5584641f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/58c6b3edec05463abfef07bf5584641f 2023-04-28 08:56:15,747 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/58c6b3edec05463abfef07bf5584641f, entries=150, sequenceid=82, filesize=25.1 K 2023-04-28 08:56:15,748 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/3c657b110dd24fc6b31e637110a9edff as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/3c657b110dd24fc6b31e637110a9edff 2023-04-28 08:56:15,755 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/3c657b110dd24fc6b31e637110a9edff, entries=150, sequenceid=82, filesize=11.6 K 2023-04-28 08:56:15,756 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/55f8d33effbb43b5b3f312f48f88ee67 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/55f8d33effbb43b5b3f312f48f88ee67 2023-04-28 08:56:15,760 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/55f8d33effbb43b5b3f312f48f88ee67, entries=150, sequenceid=82, filesize=11.6 K 2023-04-28 08:56:15,760 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 92c5373160b2e33e00d0bc96891cd0cb in 882ms, sequenceid=82, compaction requested=true 2023-04-28 08:56:15,760 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:15,760 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:15,760 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:15,760 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:15,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:15,761 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:15,762 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:15,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:15,763 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:15,763 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,763 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:15,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:15,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8b2bbf[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:15,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:15,764 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,765 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:15,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:15,765 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:15,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,765 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:15,767 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:15,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,767 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:15,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:15,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:15,767 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:15,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:15,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:15,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:15,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:15,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:15,770 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:15,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:15,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:15,770 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:15,770 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:15,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,772 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:15,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:15,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:15,772 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,773 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:15,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:15,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,776 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:15,776 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:15,776 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:15,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:15,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:15,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:15,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:56:15,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:15,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:15,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:15,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:15,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:15,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:15,796 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d5396021ceee49d99750eb20bb0b45e7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d5396021ceee49d99750eb20bb0b45e7 2023-04-28 08:56:15,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672235799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672235799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672235800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672235802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,876 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:15,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:15,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:15,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:15,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:15,880 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:15,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:15,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:15,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:15,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,880 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:15,881 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:15,881 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:15,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:15,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,882 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:15,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:15,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:15,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:15,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:15,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:15,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:15,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:15,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:15,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:15,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:15,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:15,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:15,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:15,884 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:15,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:15,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:15,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:15,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:15,886 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,886 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:15,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,887 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:15,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:15,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4726d04b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:15,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:15,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,889 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:15,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:15,889 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:15,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:15,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:15,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:15,891 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:15,891 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:15,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:15,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:15,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:15,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:15,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:15,894 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:15,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:15,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:15,894 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:15,894 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:15,896 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,896 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,896 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:15,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:15,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:15,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:15,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:15,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,898 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:15,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:15,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:15,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:15,901 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:15,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:15,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:15,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:15,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:15,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:15,902 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:15,902 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:15,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:15,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:15,902 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:15,902 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:15,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672235903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672235903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,906 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672235904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:15,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:15,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672235906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,002 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:16,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:16,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:16,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:16,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:16,006 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:16,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:16,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:16,007 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,007 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,008 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,008 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:16,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:16,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:16,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:16,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:16,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:16,010 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:16,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:16,012 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,013 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,013 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,013 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:16,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc5fed5[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:16,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:16,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,022 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:16,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:16,022 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:16,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:16,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:16,023 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,023 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:16,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:16,026 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:16,026 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:16,026 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:16,026 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:16,026 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:16,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,027 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,030 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,032 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,032 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,032 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,032 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,032 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:16,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:16,032 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:16,033 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,033 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,033 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:16,034 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:16,033 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672236106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672236106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672236109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672236110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,134 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:16,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:16,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:16,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:16,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:16,137 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:16,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:16,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:16,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,140 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,140 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:16,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:16,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:16,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:16,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:16,142 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,142 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:16,142 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:16,142 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:16,142 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:16,144 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,144 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:16,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25d433d4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:16,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:16,146 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,147 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:16,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:16,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:16,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,147 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:16,148 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,148 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:16,148 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,148 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:16,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:16,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:16,151 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:16,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:16,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:16,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:16,152 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:16,154 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,154 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,154 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,155 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,156 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:16,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:16,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,159 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,159 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,159 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,159 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,159 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,159 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:16,159 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:16,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,202 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/5ff4d96e1e684e868e9f56079f0c4383 2023-04-28 08:56:16,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/32ca0760350b4324b6ec35b55bfe137f 2023-04-28 08:56:16,259 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:16,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:16,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:16,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:16,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:16,263 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:16,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:16,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:16,263 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,264 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,265 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,265 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:16,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:16,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:16,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:16,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:16,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:16,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:16,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:16,266 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:16,267 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,267 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:16,267 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:16,267 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,267 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:16,267 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:16,267 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,267 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:16,267 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:16,269 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,269 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,269 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,269 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:16,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13343202[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:16,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:16,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,273 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,273 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,273 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:16,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:16,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:16,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:16,275 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,275 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:16,275 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,275 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:16,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:16,278 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:16,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:16,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:16,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:16,278 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:16,279 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,279 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,279 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,281 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,283 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,283 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,283 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,284 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,284 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:16,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:16,284 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,284 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,284 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,284 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,284 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:16,284 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,284 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:16,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:16,385 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:16,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:16,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:16,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:16,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:16,389 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:16,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:16,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:16,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,390 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,392 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,392 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,392 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:16,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:16,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:16,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:16,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:16,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:16,393 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:16,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:16,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:16,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:16,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:16,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:16,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:16,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62611ca7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:16,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:16,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,399 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:16,399 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:16,399 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:16,399 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,399 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:16,401 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:16,401 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,401 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,401 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:16,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:16,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:16,403 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:16,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:16,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:16,404 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:16,404 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:16,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,422 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672236410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672236411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672236422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:16,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:16,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:16,424 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:16,425 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:16,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,424 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,425 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,425 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672236422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,525 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:16,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:16,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:16,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:16,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:16,528 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:16,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:16,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:16,529 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,529 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,530 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:16,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:16,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:16,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:16,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:16,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:16,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:16,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:16,532 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,532 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:16,532 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:16,532 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:16,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:16,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:16,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:16,533 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,534 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:16,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:16,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40706568[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:16,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:16,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:16,537 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,537 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:16,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:16,537 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:16,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,539 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:16,539 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,539 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:16,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:16,542 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:16,542 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:16,542 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:16,542 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:16,542 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:16,543 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,543 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:16,543 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:16,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:16,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:16,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,548 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:16,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:16,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,548 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:16,549 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,549 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:16,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:16,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:16,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:16,549 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:16,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/891a3fe21dba408caf3326f7028e8964 2023-04-28 08:56:16,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/5ff4d96e1e684e868e9f56079f0c4383 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/5ff4d96e1e684e868e9f56079f0c4383 2023-04-28 08:56:16,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/5ff4d96e1e684e868e9f56079f0c4383, entries=200, sequenceid=95, filesize=31.9 K 2023-04-28 08:56:16,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/32ca0760350b4324b6ec35b55bfe137f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32ca0760350b4324b6ec35b55bfe137f 2023-04-28 08:56:16,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32ca0760350b4324b6ec35b55bfe137f, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 08:56:16,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/891a3fe21dba408caf3326f7028e8964 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/891a3fe21dba408caf3326f7028e8964 2023-04-28 08:56:16,641 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/891a3fe21dba408caf3326f7028e8964, entries=150, sequenceid=95, filesize=11.6 K 2023-04-28 08:56:16,642 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 92c5373160b2e33e00d0bc96891cd0cb in 860ms, sequenceid=95, compaction requested=true 2023-04-28 08:56:16,642 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:16,642 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:16,642 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:56:16,642 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:56:16,642 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:16,642 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:16,643 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 142610 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:56:16,644 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:16,644 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,644 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e9562ce0b8644a8b9092181f1788cc57, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a45f472eef0248d7b186b74a028dedb1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c2f60298b6a41eaba47871ed243eb2d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/58c6b3edec05463abfef07bf5584641f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/5ff4d96e1e684e868e9f56079f0c4383] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=139.3 K 2023-04-28 08:56:16,645 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:56:16,645 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:16,645 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,645 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/eee7e57bc5564e318300c22e18e63594, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/633955da885a4f15858e7a3fa8afda41, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ee388e0d93d74faa92e2b5e94707e238, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/3c657b110dd24fc6b31e637110a9edff, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32ca0760350b4324b6ec35b55bfe137f] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=58.0 K 2023-04-28 08:56:16,645 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e9562ce0b8644a8b9092181f1788cc57, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682672172256 2023-04-28 08:56:16,646 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting eee7e57bc5564e318300c22e18e63594, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682672172256 2023-04-28 08:56:16,646 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 633955da885a4f15858e7a3fa8afda41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672172291 2023-04-28 08:56:16,646 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a45f472eef0248d7b186b74a028dedb1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672172291 2023-04-28 08:56:16,646 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ee388e0d93d74faa92e2b5e94707e238, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682672172918 2023-04-28 08:56:16,647 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1c2f60298b6a41eaba47871ed243eb2d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682672172918 2023-04-28 08:56:16,647 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 3c657b110dd24fc6b31e637110a9edff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682672173544 2023-04-28 08:56:16,647 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 58c6b3edec05463abfef07bf5584641f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682672173544 2023-04-28 08:56:16,647 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 32ca0760350b4324b6ec35b55bfe137f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672175674 2023-04-28 08:56:16,647 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 5ff4d96e1e684e868e9f56079f0c4383, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672175674 2023-04-28 08:56:16,649 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:16,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:16,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:16,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:16,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:16,653 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:16,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:16,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:16,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,656 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:16,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:16,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:16,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:16,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:16,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:16,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:16,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:16,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:16,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:16,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:16,662 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:16,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:16,663 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-28 08:56:16,663 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,663 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:16,663 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:16,665 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#552 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:16,684 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#551 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:16,723 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/1f22ffc198244fa0aa21649ea06dd83b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f22ffc198244fa0aa21649ea06dd83b 2023-04-28 08:56:16,728 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into 1f22ffc198244fa0aa21649ea06dd83b(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:16,729 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:16,729 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=11, startTime=1682672176642; duration=0sec 2023-04-28 08:56:16,729 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:16,729 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-28 08:56:16,730 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-28 08:56:16,730 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/C is initiating minor compaction (all files) 2023-04-28 08:56:16,731 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/C in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:16,731 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a07fc6e7427d4db88791253fac3d8cff, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a2e3c085cfa5403d8894c47b58bd5bfe, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/5a521c2f67564c0484e8af6ac6420c60, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/55f8d33effbb43b5b3f312f48f88ee67, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/891a3fe21dba408caf3326f7028e8964] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=58.0 K 2023-04-28 08:56:16,731 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a07fc6e7427d4db88791253fac3d8cff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1682672172256 2023-04-28 08:56:16,731 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a2e3c085cfa5403d8894c47b58bd5bfe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1682672172291 2023-04-28 08:56:16,731 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 5a521c2f67564c0484e8af6ac6420c60, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1682672172918 2023-04-28 08:56:16,732 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 55f8d33effbb43b5b3f312f48f88ee67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682672173544 2023-04-28 08:56:16,732 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 891a3fe21dba408caf3326f7028e8964, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672175674 2023-04-28 08:56:16,739 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#C#compaction#554 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:16,746 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a3e062f8dc804242a2ede0939cdde2ac as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a3e062f8dc804242a2ede0939cdde2ac 2023-04-28 08:56:16,750 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/C of 92c5373160b2e33e00d0bc96891cd0cb into a3e062f8dc804242a2ede0939cdde2ac(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:16,750 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:16,750 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/C, priority=11, startTime=1682672176642; duration=0sec 2023-04-28 08:56:16,750 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:16,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:16,927 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:16,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672236937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672236937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672236937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:16,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:16,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672236939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672237040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672237040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672237040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672237041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,091 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/1db82857eeff4529b4e478cf39083cf4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/1db82857eeff4529b4e478cf39083cf4 2023-04-28 08:56:17,095 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into 1db82857eeff4529b4e478cf39083cf4(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:17,095 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:17,095 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=11, startTime=1682672176642; duration=0sec 2023-04-28 08:56:17,095 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:17,107 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e0a91b67cb554e159237e01ddb27dd78 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e0a91b67cb554e159237e01ddb27dd78 2023-04-28 08:56:17,118 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=114, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/687ad650ccd04e7b9e1a5309657c53fa 2023-04-28 08:56:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672237242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672237243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672237243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672237243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,533 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/0f96e79fcd3743d79363d79c779d547e 2023-04-28 08:56:17,542 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=114 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/4370db10f288459694e875321423d884 2023-04-28 08:56:17,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672237545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,546 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/687ad650ccd04e7b9e1a5309657c53fa as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/687ad650ccd04e7b9e1a5309657c53fa 2023-04-28 08:56:17,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672237546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672237547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672237547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,550 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/687ad650ccd04e7b9e1a5309657c53fa, entries=150, sequenceid=114, filesize=25.1 K 2023-04-28 08:56:17,550 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/0f96e79fcd3743d79363d79c779d547e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0f96e79fcd3743d79363d79c779d547e 2023-04-28 08:56:17,554 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0f96e79fcd3743d79363d79c779d547e, entries=150, sequenceid=114, filesize=11.6 K 2023-04-28 08:56:17,555 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/4370db10f288459694e875321423d884 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4370db10f288459694e875321423d884 2023-04-28 08:56:17,558 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4370db10f288459694e875321423d884, entries=150, sequenceid=114, filesize=11.6 K 2023-04-28 08:56:17,558 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 92c5373160b2e33e00d0bc96891cd0cb in 895ms, sequenceid=114, compaction requested=false 2023-04-28 08:56:17,558 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:17,558 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:17,558 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:17,558 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:17,558 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:17,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:17,564 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:17,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:17,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:17,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44bea421[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:17,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:17,566 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,567 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,567 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:17,567 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:17,567 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:17,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:17,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:17,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:17,568 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:17,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:17,571 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:17,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:17,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:17,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:17,571 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:17,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:17,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:17,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:17,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:17,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:17,579 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:17,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:17,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:17,579 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,579 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:17,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:17,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:17,579 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:17,680 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:17,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:17,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:17,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:17,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:17,683 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:17,683 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:17,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:17,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:17,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:17,685 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:17,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:17,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,686 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:17,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:17,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:17,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:17,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:17,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:17,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:17,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:17,691 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:17,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:17,692 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:17,692 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:17,692 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:17,692 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-28 08:56:17,693 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:17,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:17,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:17,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:17,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:17,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:17,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:17,693 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:17,726 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304283be9c008db5a4394b09c32bcde8de36d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283be9c008db5a4394b09c32bcde8de36d 2023-04-28 08:56:17,732 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/cba75afd865b40ada791221d889420e2 2023-04-28 08:56:17,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672237744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:17,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:17,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672237847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:18,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672238048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:18,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59128 deadline: 1682672238050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:18,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672238050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:18,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672238050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:18,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672238051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,144 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d816731bbeb5467a8bec8d349cfbb5bd 2023-04-28 08:56:18,155 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/653990d207e84e6081aab134b9bf4f87 2023-04-28 08:56:18,159 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/cba75afd865b40ada791221d889420e2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/cba75afd865b40ada791221d889420e2 2023-04-28 08:56:18,163 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/cba75afd865b40ada791221d889420e2, entries=200, sequenceid=131, filesize=32.0 K 2023-04-28 08:56:18,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d816731bbeb5467a8bec8d349cfbb5bd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d816731bbeb5467a8bec8d349cfbb5bd 2023-04-28 08:56:18,167 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d816731bbeb5467a8bec8d349cfbb5bd, entries=150, sequenceid=131, filesize=11.7 K 2023-04-28 08:56:18,168 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/653990d207e84e6081aab134b9bf4f87 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/653990d207e84e6081aab134b9bf4f87 2023-04-28 08:56:18,172 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/653990d207e84e6081aab134b9bf4f87, entries=150, sequenceid=131, filesize=11.7 K 2023-04-28 08:56:18,173 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 92c5373160b2e33e00d0bc96891cd0cb in 481ms, sequenceid=131, compaction requested=true 2023-04-28 08:56:18,173 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:18,173 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:18,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:18,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:18,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:18,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:18,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:18,174 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:18,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:18,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:18,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:18,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a82a96[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:18,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:18,177 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:18,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:18,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:18,180 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,180 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:18,181 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:18,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:18,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:18,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:18,184 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:18,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:18,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:18,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:18,184 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:18,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:18,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:18,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:18,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:18,188 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:18,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,194 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:18,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:18,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:18,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:18,194 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:18,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:18,194 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:18,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:18,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,195 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,195 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:18,195 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,195 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,294 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:18,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:18,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:18,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:18,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:18,298 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:18,298 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:18,298 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:18,298 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:18,298 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:18,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:18,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,300 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:18,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:18,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:18,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:18,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:18,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:18,302 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:18,302 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:18,302 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:18,302 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:18,319 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d0892459e2ee46c899834fb55c2dac00 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d0892459e2ee46c899834fb55c2dac00 2023-04-28 08:56:18,325 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/2b3d2b31e6854cabbf3eb98a5e9620a1 2023-04-28 08:56:18,335 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/2b60a63b5da940c28c3205710a37660e 2023-04-28 08:56:18,345 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/4dc55ea28e6f4d4ba52fd2836bef56bc 2023-04-28 08:56:18,348 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/2b3d2b31e6854cabbf3eb98a5e9620a1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2b3d2b31e6854cabbf3eb98a5e9620a1 2023-04-28 08:56:18,352 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2b3d2b31e6854cabbf3eb98a5e9620a1, entries=150, sequenceid=149, filesize=25.3 K 2023-04-28 08:56:18,353 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/2b60a63b5da940c28c3205710a37660e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b60a63b5da940c28c3205710a37660e 2023-04-28 08:56:18,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:18,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:18,356 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b60a63b5da940c28c3205710a37660e, entries=150, sequenceid=149, filesize=11.8 K 2023-04-28 08:56:18,357 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/4dc55ea28e6f4d4ba52fd2836bef56bc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4dc55ea28e6f4d4ba52fd2836bef56bc 2023-04-28 08:56:18,361 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4dc55ea28e6f4d4ba52fd2836bef56bc, entries=150, sequenceid=149, filesize=11.8 K 2023-04-28 08:56:18,362 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=6.71 KB/6870 for 92c5373160b2e33e00d0bc96891cd0cb in 60ms, sequenceid=149, compaction requested=true 2023-04-28 08:56:18,362 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:18,362 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:18,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:18,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:18,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:18,362 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:18,376 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,376 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:18,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:18,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:18,377 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:18,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:18,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc694c0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:18,378 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:18,379 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,380 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:18,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:18,380 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:18,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,380 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:18,382 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:18,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:18,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:18,382 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:18,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:18,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:18,385 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:18,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:18,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:18,385 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:18,385 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:18,387 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,387 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,387 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:18,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:18,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:18,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:18,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,391 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:18,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:18,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:18,391 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:18,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,392 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:18,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:18,392 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:18,392 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:18,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:18,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:18,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:18,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:18,492 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:18,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:18,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:18,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:18,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:18,496 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:18,496 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:18,496 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:18,496 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:18,497 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,497 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,498 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:18,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:18,498 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,499 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:18,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:18,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:18,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:18,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:18,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:18,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:18,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:18,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:18,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:18,503 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:18,503 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:18,504 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:18,504 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:56:18,504 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:18,504 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:18,504 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:18,504 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:18,504 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:18,504 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:18,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:18,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:18,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672178557, max timestamp=1682672178576Num uniques 859; 2023-04-28 08:56:18,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672178557, max timestamp=1682672178576Num uniques 859; 2023-04-28 08:56:18,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672178557, max timestamp=1682672178576Num uniques 859; 2023-04-28 08:56:18,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:18,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:18,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:18,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682672178625, max timestamp=1682672178650Num uniques 1018; 2023-04-28 08:56:18,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682672178625, max timestamp=1682672178650Num uniques 1018; 2023-04-28 08:56:18,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 08:56:18,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1682672178625, max timestamp=1682672178650Num uniques 1018; 2023-04-28 08:56:18,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 08:56:18,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-28 08:56:18,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672178698, max timestamp=1682672178720Num uniques 1027; 2023-04-28 08:56:18,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 08:56:18,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672178698, max timestamp=1682672178720Num uniques 1027; 2023-04-28 08:56:18,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 08:56:18,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672178698, max timestamp=1682672178720Num uniques 1027; 2023-04-28 08:56:18,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-28 08:56:18,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:18,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672238786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:18,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672238889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:18,946 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ff508cb864dd4edab998bfa9cee1b400 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ff508cb864dd4edab998bfa9cee1b400 2023-04-28 08:56:19,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672239051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672239055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672239058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672239059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672239093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,352 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/1c296b8c3e284f8581637746304836c8 2023-04-28 08:56:19,363 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/32bcf56732874b4da68ebc1c345838ac 2023-04-28 08:56:19,375 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/1430f9d6e224434dbab86befc84d6ea7 2023-04-28 08:56:19,380 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/1c296b8c3e284f8581637746304836c8 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c296b8c3e284f8581637746304836c8 2023-04-28 08:56:19,384 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c296b8c3e284f8581637746304836c8, entries=100, sequenceid=158, filesize=18.5 K 2023-04-28 08:56:19,385 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/32bcf56732874b4da68ebc1c345838ac as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32bcf56732874b4da68ebc1c345838ac 2023-04-28 08:56:19,388 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32bcf56732874b4da68ebc1c345838ac, entries=50, sequenceid=158, filesize=7.1 K 2023-04-28 08:56:19,389 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/1430f9d6e224434dbab86befc84d6ea7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/1430f9d6e224434dbab86befc84d6ea7 2023-04-28 08:56:19,393 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/1430f9d6e224434dbab86befc84d6ea7, entries=50, sequenceid=158, filesize=7.1 K 2023-04-28 08:56:19,394 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 92c5373160b2e33e00d0bc96891cd0cb in 890ms, sequenceid=158, compaction requested=true 2023-04-28 08:56:19,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:19,394 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:19,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:19,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:19,394 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:19,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,397 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,397 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:19,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fdee915[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:19,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:19,398 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,399 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,399 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:19,399 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:19,399 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:19,399 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,400 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:19,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:19,400 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=147.60 KB heapSize=419.16 KB 2023-04-28 08:56:19,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:19,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:56:19,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:19,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:56:19,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:19,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:56:19,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:19,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:19,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:19,407 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:19,407 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:19,407 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:19,407 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:19,407 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:19,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,409 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,412 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,415 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428eab178aacca34872a1a0353c85308e8d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428eab178aacca34872a1a0353c85308e8d 2023-04-28 08:56:19,416 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,416 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,416 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,416 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:19,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:19,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,417 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:19,416 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,416 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,417 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,417 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,417 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,417 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:19,417 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:19,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,421 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/20f0f87e917d478ab3cb146748cd92c5 2023-04-28 08:56:19,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672239425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/5e225f00314640aba7b704cfe2d44407 2023-04-28 08:56:19,518 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:19,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:19,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:19,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:19,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:19,521 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:19,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:19,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:19,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,522 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,523 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,523 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:19,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:19,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:19,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:19,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:19,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:19,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:19,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:19,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:19,524 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,524 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:19,524 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:19,525 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:19,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:19,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:19,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:19,526 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,526 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,526 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,526 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:19,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:19,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ef0d163[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:19,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:19,529 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,530 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,530 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:19,530 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:19,530 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:19,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:19,532 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:19,532 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672239531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:19,536 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:19,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:19,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:19,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:19,536 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:19,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,538 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,545 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:19,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:19,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,545 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,546 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:19,546 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:19,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,646 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:19,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:19,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:19,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:19,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:19,651 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:19,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:19,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:19,652 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,652 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,654 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,655 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:19,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:19,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:19,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:19,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:19,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:19,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:19,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,658 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:19,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59bbaab1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:19,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:19,660 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,662 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:19,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:19,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:19,662 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,662 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:19,663 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:19,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,663 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:19,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:19,665 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:19,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:19,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:19,666 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:19,666 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:19,668 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,668 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,668 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,669 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,670 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,672 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:19,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:19,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,672 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:19,672 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:19,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:19,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:19,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672239735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,772 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:19,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:19,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:19,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:19,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:19,776 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:19,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:19,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:19,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,776 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,778 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,778 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,778 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,778 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:19,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:19,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:19,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:19,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:19,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:19,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:19,779 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:19,780 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,780 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:19,780 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:19,780 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,780 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:19,780 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:19,780 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,780 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:19,780 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:19,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,781 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,781 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:19,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71f355d2[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:19,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:19,783 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,785 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:19,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:19,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:19,785 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,785 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:19,786 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:19,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,786 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,786 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:19,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:19,788 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:19,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:19,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:19,788 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:19,789 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:19,790 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,790 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,790 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,791 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:19,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:19,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:19,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,795 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,795 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:19,795 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:19,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:19,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/db3400cacc134c81b6cf63c367277c23 2023-04-28 08:56:19,843 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/20f0f87e917d478ab3cb146748cd92c5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/20f0f87e917d478ab3cb146748cd92c5 2023-04-28 08:56:19,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/20f0f87e917d478ab3cb146748cd92c5, entries=150, sequenceid=230, filesize=25.3 K 2023-04-28 08:56:19,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/5e225f00314640aba7b704cfe2d44407 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5e225f00314640aba7b704cfe2d44407 2023-04-28 08:56:19,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5e225f00314640aba7b704cfe2d44407, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:56:19,852 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/db3400cacc134c81b6cf63c367277c23 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/db3400cacc134c81b6cf63c367277c23 2023-04-28 08:56:19,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/db3400cacc134c81b6cf63c367277c23, entries=150, sequenceid=230, filesize=11.8 K 2023-04-28 08:56:19,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.11 KB/429168, currentSize=33.54 KB/34350 for 92c5373160b2e33e00d0bc96891cd0cb in 456ms, sequenceid=230, compaction requested=true 2023-04-28 08:56:19,856 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:19,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:19,856 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:56:19,856 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:56:19,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:19,856 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:19,859 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155088 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:56:19,859 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:19,859 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67202 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:56:19,859 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,859 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:19,859 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,859 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f22ffc198244fa0aa21649ea06dd83b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/687ad650ccd04e7b9e1a5309657c53fa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/cba75afd865b40ada791221d889420e2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2b3d2b31e6854cabbf3eb98a5e9620a1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c296b8c3e284f8581637746304836c8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/20f0f87e917d478ab3cb146748cd92c5] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=151.5 K 2023-04-28 08:56:19,859 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/1db82857eeff4529b4e478cf39083cf4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0f96e79fcd3743d79363d79c779d547e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d816731bbeb5467a8bec8d349cfbb5bd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b60a63b5da940c28c3205710a37660e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32bcf56732874b4da68ebc1c345838ac, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5e225f00314640aba7b704cfe2d44407] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=65.6 K 2023-04-28 08:56:19,859 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1f22ffc198244fa0aa21649ea06dd83b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672175674 2023-04-28 08:56:19,863 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 1db82857eeff4529b4e478cf39083cf4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672175674 2023-04-28 08:56:19,871 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0f96e79fcd3743d79363d79c779d547e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1682672175786 2023-04-28 08:56:19,880 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d816731bbeb5467a8bec8d349cfbb5bd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682672176934 2023-04-28 08:56:19,880 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 687ad650ccd04e7b9e1a5309657c53fa, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1682672175786 2023-04-28 08:56:19,880 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2b60a63b5da940c28c3205710a37660e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672177736 2023-04-28 08:56:19,881 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting cba75afd865b40ada791221d889420e2, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682672176934 2023-04-28 08:56:19,881 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 32bcf56732874b4da68ebc1c345838ac, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672178478 2023-04-28 08:56:19,881 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 2b3d2b31e6854cabbf3eb98a5e9620a1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672177736 2023-04-28 08:56:19,881 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5e225f00314640aba7b704cfe2d44407, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672178781 2023-04-28 08:56:19,882 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1c296b8c3e284f8581637746304836c8, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672178472 2023-04-28 08:56:19,883 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 20f0f87e917d478ab3cb146748cd92c5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672178781 2023-04-28 08:56:19,893 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#569 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:19,895 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:19,897 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#570 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:19,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:19,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:19,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:19,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:19,903 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:19,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:19,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:19,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:19,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:19,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:19,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:19,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:19,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:19,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:19,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:19,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:19,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:19,919 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:19,919 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,920 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:19,920 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 08:56:19,920 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:19,920 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:19,920 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:19,920 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:19,920 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:19,920 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:19,951 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304282e42e31effc04938a14e20501f03928a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282e42e31effc04938a14e20501f03928a 2023-04-28 08:56:19,956 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ad795c18743245999ed11bdd68bf395c 2023-04-28 08:56:19,965 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/502ce08c06f84c17a102886998339ad4 2023-04-28 08:56:19,973 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/638df471229c45c4adb03ec68611537b 2023-04-28 08:56:19,977 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ad795c18743245999ed11bdd68bf395c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ad795c18743245999ed11bdd68bf395c 2023-04-28 08:56:19,982 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ad795c18743245999ed11bdd68bf395c, entries=100, sequenceid=238, filesize=18.5 K 2023-04-28 08:56:19,985 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/502ce08c06f84c17a102886998339ad4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/502ce08c06f84c17a102886998339ad4 2023-04-28 08:56:19,989 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/502ce08c06f84c17a102886998339ad4, entries=100, sequenceid=238, filesize=9.4 K 2023-04-28 08:56:19,990 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/638df471229c45c4adb03ec68611537b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/638df471229c45c4adb03ec68611537b 2023-04-28 08:56:19,993 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/638df471229c45c4adb03ec68611537b, entries=100, sequenceid=238, filesize=9.4 K 2023-04-28 08:56:19,994 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 92c5373160b2e33e00d0bc96891cd0cb in 74ms, sequenceid=238, compaction requested=true 2023-04-28 08:56:19,994 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:19,994 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:19,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:19,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:19,994 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:19,995 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:19,996 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,996 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:19,996 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,996 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:19,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:19,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59cd8cb8[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:19,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:19,998 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,999 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:19,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:19,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:19,999 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,000 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,001 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,001 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,004 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,004 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,004 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,007 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,007 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,013 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:20,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,013 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,014 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,014 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,114 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:20,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:20,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:20,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:20,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:20,119 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:20,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:20,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:20,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,119 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,120 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,120 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,121 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,121 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:20,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:20,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:20,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:20,123 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:56:20,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:20,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,123 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:20,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:20,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:20,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:20,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:20,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:20,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:20,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:20,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:20,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:20,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:20,127 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:20,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:20,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:20,127 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:20,133 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,133 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:20,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:20,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31de252b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:20,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:20,135 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,137 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:20,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:20,137 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,137 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,138 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,138 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,138 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,138 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,150 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428b5743756907c4da3bea76726b9f5a7d2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5743756907c4da3bea76726b9f5a7d2 2023-04-28 08:56:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,154 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,154 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,154 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,154 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,154 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,156 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,156 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,156 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,157 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,158 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,166 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,166 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672180187, max timestamp=1682672180207Num uniques 859; 2023-04-28 08:56:20,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672180187, max timestamp=1682672180207Num uniques 859; 2023-04-28 08:56:20,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:20,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672180187, max timestamp=1682672180207Num uniques 859; 2023-04-28 08:56:20,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:20,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:20,266 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:20,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:20,270 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:20,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:20,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:20,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,270 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,271 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,271 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,272 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,272 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:20,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:20,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:20,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:20,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:20,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:20,273 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:20,273 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,273 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:20,273 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:20,273 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,274 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:20,274 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:20,274 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,274 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:20,274 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:20,275 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,275 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,275 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,275 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35b4ff83[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:20,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:20,277 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,278 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,278 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:20,278 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:20,278 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,280 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,280 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,283 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,283 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,285 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,285 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,285 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,287 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,289 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,289 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:20,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,289 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,289 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672240296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,321 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/ba3c356e3686423da48372a84db5c2a5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ba3c356e3686423da48372a84db5c2a5 2023-04-28 08:56:20,325 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into ba3c356e3686423da48372a84db5c2a5(size=12.1 K), total size for store is 21.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:20,325 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:20,325 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=10, startTime=1682672179856; duration=0sec 2023-04-28 08:56:20,325 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:20,325 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-28 08:56:20,327 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 76844 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-28 08:56:20,327 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/C is initiating minor compaction (all files) 2023-04-28 08:56:20,327 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/C in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,328 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a3e062f8dc804242a2ede0939cdde2ac, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4370db10f288459694e875321423d884, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/653990d207e84e6081aab134b9bf4f87, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4dc55ea28e6f4d4ba52fd2836bef56bc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/1430f9d6e224434dbab86befc84d6ea7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/db3400cacc134c81b6cf63c367277c23, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/638df471229c45c4adb03ec68611537b] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=75.0 K 2023-04-28 08:56:20,328 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a3e062f8dc804242a2ede0939cdde2ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1682672175674 2023-04-28 08:56:20,328 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4370db10f288459694e875321423d884, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=114, earliestPutTs=1682672175786 2023-04-28 08:56:20,328 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 653990d207e84e6081aab134b9bf4f87, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1682672176934 2023-04-28 08:56:20,329 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 4dc55ea28e6f4d4ba52fd2836bef56bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1682672177736 2023-04-28 08:56:20,329 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 1430f9d6e224434dbab86befc84d6ea7, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1682672178478 2023-04-28 08:56:20,329 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting db3400cacc134c81b6cf63c367277c23, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672178781 2023-04-28 08:56:20,330 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 638df471229c45c4adb03ec68611537b, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682672179420 2023-04-28 08:56:20,340 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#C#compaction#575 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:20,343 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/c73f830010c449a7ae4e941c58ac6522 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/c73f830010c449a7ae4e941c58ac6522 2023-04-28 08:56:20,348 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into c73f830010c449a7ae4e941c58ac6522(size=25.7 K), total size for store is 44.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:20,348 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:20,348 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=10, startTime=1682672179856; duration=0sec 2023-04-28 08:56:20,348 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:20,389 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:20,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:20,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:20,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:20,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:20,393 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:20,393 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,393 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:20,393 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:20,393 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,395 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,395 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:20,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:20,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:20,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:20,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:20,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:20,396 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:20,396 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,396 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:20,396 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:20,397 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,397 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:20,397 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:20,397 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,397 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:20,397 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:20,400 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,400 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,400 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,400 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e026122[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:20,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:20,402 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:20,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:20,403 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:20,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672240400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,404 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,404 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,404 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,404 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,407 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,407 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,407 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,407 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,407 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,408 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,409 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,414 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,414 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,415 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,415 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:20,515 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:20,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:20,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:20,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:20,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:20,518 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:20,518 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,518 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:20,518 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:20,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,519 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,520 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,520 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,521 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:20,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:20,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:20,522 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,522 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:20,522 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:20,522 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:20,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:20,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1f9e16[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:20,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:20,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,531 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:20,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:20,531 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,531 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,531 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,532 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,533 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,535 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,536 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,537 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,538 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,539 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,542 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:20,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,542 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,543 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,543 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,565 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=248, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/a68d4e6dd8d6441b8dcf7081100439e6 2023-04-28 08:56:20,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:20,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672240606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,643 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:20,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:20,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:20,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:20,647 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:20,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:20,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:20,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,647 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,650 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,650 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,651 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:20,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:20,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:20,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:20,652 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,652 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:20,652 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:20,652 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,652 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:20,653 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:20,654 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,654 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,654 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,654 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba81923[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:20,656 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:20,656 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,657 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,657 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:20,657 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:20,657 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,657 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,658 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,658 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,664 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,664 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,664 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,667 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,667 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,667 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,667 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,668 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,671 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,672 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:20,672 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,672 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,749 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/52a7cfe9d68e49bbaa50937e234ca97f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52a7cfe9d68e49bbaa50937e234ca97f 2023-04-28 08:56:20,754 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/C of 92c5373160b2e33e00d0bc96891cd0cb into 52a7cfe9d68e49bbaa50937e234ca97f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:20,754 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:20,754 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/C, priority=9, startTime=1682672179856; duration=0sec 2023-04-28 08:56:20,754 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:20,772 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:20,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:20,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:20,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:20,779 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:20,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:20,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:20,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,779 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,780 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,780 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,781 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,781 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:20,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:20,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:20,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:20,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:20,782 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:20,782 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,782 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:20,783 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:20,783 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:20,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:20,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:20,783 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:20,784 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,784 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,785 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3972b332[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:20,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:20,787 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,789 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:20,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:20,789 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,789 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,789 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,790 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,793 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,793 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,793 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,793 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,793 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,795 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,796 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,799 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,799 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,799 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,799 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,900 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:20,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:20,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:20,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:20,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:20,903 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:20,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:20,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:20,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,904 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,907 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:20,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:20,908 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:20,908 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:20,908 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:20,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:20,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:20,909 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,909 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:20,909 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:20,909 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:20,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:20,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:20,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:20,909 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:20,911 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,911 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:20,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672240909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,913 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:20,913 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74b4de4f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:20,914 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:20,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:20,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:20,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:20,920 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,920 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:20,920 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,920 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:20,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:20,922 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:20,922 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:20,922 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:20,922 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:20,922 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:20,924 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,924 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:20,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,924 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:20,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,924 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:20,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:20,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:20,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:20,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:20,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:20,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:20,928 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:20,928 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:20,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:20,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:20,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:20,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:20,930 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:20,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:20,930 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:20,930 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:20,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/e35156735c6b4cea9f9de4c2132b17cb 2023-04-28 08:56:20,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/ad457d54204e412a929b4911872d5eae 2023-04-28 08:56:20,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/a68d4e6dd8d6441b8dcf7081100439e6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a68d4e6dd8d6441b8dcf7081100439e6 2023-04-28 08:56:20,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a68d4e6dd8d6441b8dcf7081100439e6, entries=200, sequenceid=248, filesize=32.1 K 2023-04-28 08:56:20,994 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/e35156735c6b4cea9f9de4c2132b17cb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/e35156735c6b4cea9f9de4c2132b17cb 2023-04-28 08:56:20,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/e35156735c6b4cea9f9de4c2132b17cb, entries=150, sequenceid=248, filesize=11.8 K 2023-04-28 08:56:20,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/ad457d54204e412a929b4911872d5eae as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ad457d54204e412a929b4911872d5eae 2023-04-28 08:56:21,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ad457d54204e412a929b4911872d5eae, entries=150, sequenceid=248, filesize=11.8 K 2023-04-28 08:56:21,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 92c5373160b2e33e00d0bc96891cd0cb in 880ms, sequenceid=248, compaction requested=true 2023-04-28 08:56:21,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:21,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:21,003 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:56:21,003 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:56:21,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:21,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:21,004 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:56:21,004 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:21,004 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:21,004 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/c73f830010c449a7ae4e941c58ac6522, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ad795c18743245999ed11bdd68bf395c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a68d4e6dd8d6441b8dcf7081100439e6] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=76.2 K 2023-04-28 08:56:21,004 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34090 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:56:21,004 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:21,004 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:21,005 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ba3c356e3686423da48372a84db5c2a5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/502ce08c06f84c17a102886998339ad4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/e35156735c6b4cea9f9de4c2132b17cb] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=33.3 K 2023-04-28 08:56:21,005 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting c73f830010c449a7ae4e941c58ac6522, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672178781 2023-04-28 08:56:21,005 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ba3c356e3686423da48372a84db5c2a5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1682672178781 2023-04-28 08:56:21,005 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ad795c18743245999ed11bdd68bf395c, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682672179420 2023-04-28 08:56:21,005 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 502ce08c06f84c17a102886998339ad4, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682672179420 2023-04-28 08:56:21,006 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a68d4e6dd8d6441b8dcf7081100439e6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682672180095 2023-04-28 08:56:21,006 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e35156735c6b4cea9f9de4c2132b17cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682672180095 2023-04-28 08:56:21,011 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#579 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:21,036 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:21,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:21,038 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#578 average throughput is 0.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:21,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:21,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:21,040 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:21,040 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:21,040 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:21,041 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:21,041 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,041 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,042 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:21,042 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:21,042 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,043 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:21,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:21,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:21,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:21,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:21,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:21,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:21,051 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/86cf192475394c88b526b282450c03bb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/86cf192475394c88b526b282450c03bb 2023-04-28 08:56:21,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:21,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:21,054 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into 86cf192475394c88b526b282450c03bb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:21,054 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:21,054 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:21,054 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=13, startTime=1682672181003; duration=0sec 2023-04-28 08:56:21,054 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:21,055 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:21,055 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-28 08:56:21,055 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-28 08:56:21,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:21,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:21,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:21,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:21,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:21,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:21,055 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-28 08:56:21,055 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-28 08:56:21,056 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. because compaction request was cancelled 2023-04-28 08:56:21,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:21,062 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:21,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672241073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672241073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,077 DEBUG [Thread-4001] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:56:21,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672241075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672241075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672241177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672241179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672241179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672241380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672241380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672241383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672241415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,483 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304281617c65665254b8ab07ebf4e67dfd182 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281617c65665254b8ab07ebf4e67dfd182 2023-04-28 08:56:21,487 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ee8dafb3a40540d59de4d5d592cec36a 2023-04-28 08:56:21,497 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/13b9e7713bd14b34a442107e75725b47 2023-04-28 08:56:21,506 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/46ccd353888c48dd834d8ca4e42e9b16 2023-04-28 08:56:21,510 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ee8dafb3a40540d59de4d5d592cec36a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ee8dafb3a40540d59de4d5d592cec36a 2023-04-28 08:56:21,513 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ee8dafb3a40540d59de4d5d592cec36a, entries=150, sequenceid=289, filesize=25.4 K 2023-04-28 08:56:21,514 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/13b9e7713bd14b34a442107e75725b47 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/13b9e7713bd14b34a442107e75725b47 2023-04-28 08:56:21,517 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/13b9e7713bd14b34a442107e75725b47, entries=150, sequenceid=289, filesize=11.9 K 2023-04-28 08:56:21,518 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/46ccd353888c48dd834d8ca4e42e9b16 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/46ccd353888c48dd834d8ca4e42e9b16 2023-04-28 08:56:21,521 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/46ccd353888c48dd834d8ca4e42e9b16, entries=150, sequenceid=289, filesize=11.9 K 2023-04-28 08:56:21,522 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 92c5373160b2e33e00d0bc96891cd0cb in 467ms, sequenceid=289, compaction requested=true 2023-04-28 08:56:21,522 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:21,522 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:21,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:21,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:21,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:21,522 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:21,524 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:21,524 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:21,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,525 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:21,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63cc11e1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:21,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:21,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,527 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:21,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:21,527 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:21,527 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,528 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:21,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,529 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:21,529 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:21,529 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:21,531 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:21,531 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:21,532 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:21,532 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:21,532 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:21,533 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,533 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:21,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,533 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:21,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:21,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:21,536 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:21,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:21,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:21,539 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:21,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:21,540 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,540 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:21,540 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:21,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:21,540 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:21,540 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,540 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:21,540 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:21,640 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:21,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:21,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:21,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:21,652 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:21,652 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:21,652 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:21,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:21,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,654 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:21,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:21,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:21,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:21,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:21,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:21,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:21,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:21,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:21,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:21,659 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:21,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:21,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:21,659 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:56:21,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:21,659 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:21,660 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:21,660 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:21,660 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:21,660 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:21,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:21,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:21,700 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428074a405464344bb38189462efc801618 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428074a405464344bb38189462efc801618 2023-04-28 08:56:21,712 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=299, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/33b77b37110347e1bfbe3c4dfb933784 2023-04-28 08:56:21,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672181705, max timestamp=1682672181715Num uniques 859; 2023-04-28 08:56:21,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:56:21,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672181705, max timestamp=1682672181715Num uniques 859; 2023-04-28 08:56:21,718 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:21,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672181705, max timestamp=1682672181715Num uniques 859; 2023-04-28 08:56:21,719 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:21,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672241741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672241742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672241744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672241845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672241847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:21,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672241848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:21,891 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/d19e26220ded4f72b3d01dc28c11acef as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/d19e26220ded4f72b3d01dc28c11acef 2023-04-28 08:56:21,896 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into d19e26220ded4f72b3d01dc28c11acef(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:21,896 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:21,896 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=13, startTime=1682672181003; duration=0sec 2023-04-28 08:56:21,896 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:22,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672242050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,052 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672242050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672242051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,123 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/cfc092d6e3fa49a6b3d2d5816dac4e1a 2023-04-28 08:56:22,145 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=299 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/47c02b2fe07740c882c79f9a3cdfdb7f 2023-04-28 08:56:22,149 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/33b77b37110347e1bfbe3c4dfb933784 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/33b77b37110347e1bfbe3c4dfb933784 2023-04-28 08:56:22,153 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/33b77b37110347e1bfbe3c4dfb933784, entries=100, sequenceid=299, filesize=18.6 K 2023-04-28 08:56:22,154 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/cfc092d6e3fa49a6b3d2d5816dac4e1a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/cfc092d6e3fa49a6b3d2d5816dac4e1a 2023-04-28 08:56:22,158 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/cfc092d6e3fa49a6b3d2d5816dac4e1a, entries=100, sequenceid=299, filesize=9.5 K 2023-04-28 08:56:22,159 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/47c02b2fe07740c882c79f9a3cdfdb7f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/47c02b2fe07740c882c79f9a3cdfdb7f 2023-04-28 08:56:22,165 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/47c02b2fe07740c882c79f9a3cdfdb7f, entries=100, sequenceid=299, filesize=9.5 K 2023-04-28 08:56:22,165 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 92c5373160b2e33e00d0bc96891cd0cb in 506ms, sequenceid=299, compaction requested=true 2023-04-28 08:56:22,165 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:22,165 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:22,165 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:22,166 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:22,166 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,166 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:22,166 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:22,167 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,167 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:22,168 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,168 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:22,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8c66dba[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:22,169 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:22,169 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:22,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:22,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:22,175 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,175 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:22,176 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:22,177 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:22,177 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:22,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:22,179 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:22,179 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:22,179 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:22,179 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:22,179 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:22,181 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,181 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,181 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:22,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,182 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:22,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,185 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,185 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:22,185 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:22,185 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:22,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:22,185 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:22,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:22,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:22,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:22,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:22,186 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:22,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,186 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:22,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:22,286 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:22,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:22,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:22,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:22,290 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:22,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:22,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:22,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:22,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:22,292 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:22,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:22,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:22,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,293 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:22,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:22,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:22,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:22,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:22,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:22,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:22,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:22,294 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:22,294 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:22,295 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:22,295 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=136.42 KB heapSize=390.06 KB 2023-04-28 08:56:22,295 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:22,295 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:22,295 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:22,295 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:22,295 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:22,295 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:22,311 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e14f06a6a2f849d78b8e237aa0d31913 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e14f06a6a2f849d78b8e237aa0d31913 2023-04-28 08:56:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:22,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:22,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672242361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672242364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672242366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672242428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672242467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672242467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672242470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672242671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672242672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672242674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,717 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/e2ccac1104f24150b06e4abfc90ab298 2023-04-28 08:56:22,727 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/2ab2ef8697c4471496855e73e68ca848 2023-04-28 08:56:22,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672242976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672242978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:22,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:22,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672242978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,138 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/671a74933ba847d0a2da3729785ac2b7 2023-04-28 08:56:23,142 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/e2ccac1104f24150b06e4abfc90ab298 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e2ccac1104f24150b06e4abfc90ab298 2023-04-28 08:56:23,146 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e2ccac1104f24150b06e4abfc90ab298, entries=150, sequenceid=337, filesize=25.4 K 2023-04-28 08:56:23,147 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/2ab2ef8697c4471496855e73e68ca848 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2ab2ef8697c4471496855e73e68ca848 2023-04-28 08:56:23,151 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2ab2ef8697c4471496855e73e68ca848, entries=150, sequenceid=337, filesize=11.9 K 2023-04-28 08:56:23,151 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/671a74933ba847d0a2da3729785ac2b7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/671a74933ba847d0a2da3729785ac2b7 2023-04-28 08:56:23,154 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/671a74933ba847d0a2da3729785ac2b7, entries=150, sequenceid=337, filesize=11.9 K 2023-04-28 08:56:23,155 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~390.02 KB/399376, currentSize=46.96 KB/48090 for 92c5373160b2e33e00d0bc96891cd0cb in 860ms, sequenceid=337, compaction requested=true 2023-04-28 08:56:23,155 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:23,155 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:23,155 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:23,155 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:23,155 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:23,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:23,157 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2401b4b3[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:23,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:23,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,160 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,160 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:23,160 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:23,160 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:23,161 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,161 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:23,162 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:23,162 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:23,162 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:23,162 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:23,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:23,165 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:23,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:23,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:23,165 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:23,165 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:23,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:23,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:23,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:23,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:23,175 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,175 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:23,176 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:23,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:23,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:23,276 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:23,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:23,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:23,280 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:23,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:23,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:23,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:23,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,284 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:23,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:23,284 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,285 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:23,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:23,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:23,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:23,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:23,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:23,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:23,287 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:23,287 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:23,287 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:56:23,288 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:23,288 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:23,288 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:23,288 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:23,288 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:23,288 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:23,304 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042892ecad97be714013878c0d91611052d9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042892ecad97be714013878c0d91611052d9 2023-04-28 08:56:23,320 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/490eb88a536c4685b9889306c0ab607b 2023-04-28 08:56:23,357 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d6af57fc34a54a2eb4257a07aebfa0e4 2023-04-28 08:56:23,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:23,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:23,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672183503, max timestamp=1682672183514Num uniques 859; 2023-04-28 08:56:23,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672183503, max timestamp=1682672183514Num uniques 859; 2023-04-28 08:56:23,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672183503, max timestamp=1682672183514Num uniques 859; 2023-04-28 08:56:23,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:56:23,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:56:23,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:56:23,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672243562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672243570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672243570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672243671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672243675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672243675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,786 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/c8d562449caa49ee86e4bb991972cb99 2023-04-28 08:56:23,791 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/490eb88a536c4685b9889306c0ab607b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/490eb88a536c4685b9889306c0ab607b 2023-04-28 08:56:23,795 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/490eb88a536c4685b9889306c0ab607b, entries=150, sequenceid=347, filesize=25.4 K 2023-04-28 08:56:23,796 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d6af57fc34a54a2eb4257a07aebfa0e4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d6af57fc34a54a2eb4257a07aebfa0e4 2023-04-28 08:56:23,800 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d6af57fc34a54a2eb4257a07aebfa0e4, entries=150, sequenceid=347, filesize=11.9 K 2023-04-28 08:56:23,801 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/c8d562449caa49ee86e4bb991972cb99 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c8d562449caa49ee86e4bb991972cb99 2023-04-28 08:56:23,804 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c8d562449caa49ee86e4bb991972cb99, entries=150, sequenceid=347, filesize=11.9 K 2023-04-28 08:56:23,805 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 92c5373160b2e33e00d0bc96891cd0cb in 518ms, sequenceid=347, compaction requested=true 2023-04-28 08:56:23,805 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:23,805 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:23,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:23,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:23,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:23,805 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:23,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:23,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@769540da[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:23,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:23,809 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,810 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:23,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:23,810 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:23,810 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,810 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:23,813 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:23,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:23,814 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:23,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:23,816 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:23,816 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:23,816 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:23,816 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:23,816 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:23,818 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,818 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,818 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:23,819 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,822 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:23,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:23,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:23,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:23,823 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:23,823 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:23,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:23,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:23,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:23,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-28 08:56:23,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:23,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:23,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:23,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:23,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:23,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:23,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672243889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672243889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672243889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,923 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:23,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:23,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:23,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:23,926 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:23,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:23,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:23,926 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:23,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,927 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,928 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:23,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:23,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:23,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:23,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:23,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:23,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:23,929 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:23,929 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:23,930 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:23,930 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:23,930 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:23,930 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:23,930 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:23,932 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,932 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:23,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:23,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755514f0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:23,934 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:23,934 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,935 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:23,935 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:23,935 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:23,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:23,936 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:23,936 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,936 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,936 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:23,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:23,939 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:23,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:23,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:23,939 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:23,939 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:23,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,940 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:23,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:23,941 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,942 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:23,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:23,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:23,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:23,945 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:23,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,945 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:23,946 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,946 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,946 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:23,946 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:23,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:23,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:23,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:23,946 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:23,946 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:23,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:23,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672243994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672243994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:23,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672243995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,046 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:24,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:24,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:24,050 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:24,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:24,050 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:24,051 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,051 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,052 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,052 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,052 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,052 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:24,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:24,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:24,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:24,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:24,054 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:24,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:24,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:24,055 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:24,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:24,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:24,055 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:24,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,056 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,057 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:24,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e451c5d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:24,058 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:24,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,060 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,060 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:24,060 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:24,060 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:24,060 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,060 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:24,069 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:24,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,069 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:24,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:24,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:24,072 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:24,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:24,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:24,072 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:24,072 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:24,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,075 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,076 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,077 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:24,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:24,080 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:24,080 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:24,079 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,079 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:24,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,080 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,180 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:24,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:24,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:24,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:24,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:24,184 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:24,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:24,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:24,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,186 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,186 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,187 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:24,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:24,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:24,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:24,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:24,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:24,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:24,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:24,189 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,189 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:24,190 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:24,190 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,190 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:24,190 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:24,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,191 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,191 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,191 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:24,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67489d0a[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:24,193 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:24,193 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,194 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,194 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:24,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:24,195 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:24,195 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,195 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:24,196 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:24,196 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,196 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,196 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:24,199 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:24,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:24,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:24,199 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:24,199 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:24,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:24,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672244200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:24,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:24,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59100 deadline: 1682672244200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672244200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,206 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,207 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,208 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,212 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,213 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:24,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:24,213 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,214 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:24,214 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:24,290 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042870bf6bd58810416ab91fc5223c68ebf1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042870bf6bd58810416ab91fc5223c68ebf1 2023-04-28 08:56:24,314 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:24,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:24,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:24,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:24,317 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:24,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:24,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:24,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,318 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,319 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,319 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,319 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:24,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:24,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:24,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:24,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:24,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:24,320 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:24,320 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,320 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:24,320 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:24,320 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:24,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:24,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:24,321 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:24,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,322 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,322 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b0782a9[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:24,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:24,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,326 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,326 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:24,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:24,326 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:24,326 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:24,328 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,328 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,328 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,328 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:24,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:24,330 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:24,330 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:24,330 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:24,331 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:24,331 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:24,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,332 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,332 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,334 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,336 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,336 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,336 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:24,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:24,337 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,337 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,337 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:24,337 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:24,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,437 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:24,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:24,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:24,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:24,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:24,441 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:24,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:24,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672244438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:24,441 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:24,443 DEBUG [Thread-4007] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:56:24,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,445 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:24,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:24,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:24,447 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,447 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:24,447 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:24,447 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:24,447 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:24,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66a8e3ad[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:24,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:24,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,452 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:24,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:24,452 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:24,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:24,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:24,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,453 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:24,456 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:24,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:24,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:24,456 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:24,456 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:24,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,458 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,459 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,461 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,463 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,464 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:24,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:24,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,464 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:24,464 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:24,464 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59132 deadline: 1682672244505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59092 deadline: 1682672244505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672244506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,564 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:24,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:24,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:24,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:24,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:24,567 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:24,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:24,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:24,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,571 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:24,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:24,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:24,573 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,573 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:24,573 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:24,573 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,573 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:24,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:24,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,575 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:24,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:24,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69bcd1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:24,576 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:24,577 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,577 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,578 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:24,578 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:24,578 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:24,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,578 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:24,580 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,580 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:24,580 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,580 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:24,582 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:24,583 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:24,583 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:24,583 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:24,583 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:24,584 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,584 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,584 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,585 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,586 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:24,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,588 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,589 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:24,589 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:24,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:24,689 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:24,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:24,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:24,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:24,692 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:24,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:24,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:24,693 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,693 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,694 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,694 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,694 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,694 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:24,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:24,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:24,695 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:24,695 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,695 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:24,695 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:24,695 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:24,696 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,696 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:24,696 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:24,696 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:24,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:24,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:24,696 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:24,696 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=384, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/8924b3f618434331a5eabb968dc2e47a 2023-04-28 08:56:24,698 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,698 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,699 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,699 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@236ad11b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:24,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:24,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,702 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,702 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:24,702 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:24,702 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:24,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,703 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:24,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,704 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:24,704 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:24,707 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:24,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:24,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:24,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:24,707 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:24,708 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/9b59721c3f2d4283bcd89b9eac5604ae 2023-04-28 08:56:24,709 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,709 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,709 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,710 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,712 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:24,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:24,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:24,716 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,716 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:24,716 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:24,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,716 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,717 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,717 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,717 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/94b6aef839644e87bec8384552b5f365 2023-04-28 08:56:24,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/8924b3f618434331a5eabb968dc2e47a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/8924b3f618434331a5eabb968dc2e47a 2023-04-28 08:56:24,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/8924b3f618434331a5eabb968dc2e47a, entries=200, sequenceid=384, filesize=32.3 K 2023-04-28 08:56:24,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/9b59721c3f2d4283bcd89b9eac5604ae as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/9b59721c3f2d4283bcd89b9eac5604ae 2023-04-28 08:56:24,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/9b59721c3f2d4283bcd89b9eac5604ae, entries=150, sequenceid=384, filesize=11.9 K 2023-04-28 08:56:24,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/94b6aef839644e87bec8384552b5f365 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94b6aef839644e87bec8384552b5f365 2023-04-28 08:56:24,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94b6aef839644e87bec8384552b5f365, entries=150, sequenceid=384, filesize=11.9 K 2023-04-28 08:56:24,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=40.25 KB/41220 for 92c5373160b2e33e00d0bc96891cd0cb in 853ms, sequenceid=384, compaction requested=true 2023-04-28 08:56:24,731 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:24,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:24,731 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:56:24,731 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:56:24,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:24,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:24,733 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156596 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:56:24,733 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:24,733 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,733 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/d19e26220ded4f72b3d01dc28c11acef, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ee8dafb3a40540d59de4d5d592cec36a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/33b77b37110347e1bfbe3c4dfb933784, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e2ccac1104f24150b06e4abfc90ab298, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/490eb88a536c4685b9889306c0ab607b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/8924b3f618434331a5eabb968dc2e47a] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=152.9 K 2023-04-28 08:56:24,733 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71000 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:56:24,733 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:24,733 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,733 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/86cf192475394c88b526b282450c03bb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/13b9e7713bd14b34a442107e75725b47, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/cfc092d6e3fa49a6b3d2d5816dac4e1a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2ab2ef8697c4471496855e73e68ca848, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d6af57fc34a54a2eb4257a07aebfa0e4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/9b59721c3f2d4283bcd89b9eac5604ae] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=69.3 K 2023-04-28 08:56:24,734 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting d19e26220ded4f72b3d01dc28c11acef, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682672180095 2023-04-28 08:56:24,734 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 86cf192475394c88b526b282450c03bb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682672180095 2023-04-28 08:56:24,734 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ee8dafb3a40540d59de4d5d592cec36a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682672180265 2023-04-28 08:56:24,734 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 13b9e7713bd14b34a442107e75725b47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682672180265 2023-04-28 08:56:24,734 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 33b77b37110347e1bfbe3c4dfb933784, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682672181073 2023-04-28 08:56:24,735 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting cfc092d6e3fa49a6b3d2d5816dac4e1a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682672181073 2023-04-28 08:56:24,735 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting e2ccac1104f24150b06e4abfc90ab298, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682672181739 2023-04-28 08:56:24,735 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2ab2ef8697c4471496855e73e68ca848, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682672181739 2023-04-28 08:56:24,735 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 490eb88a536c4685b9889306c0ab607b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682672182360 2023-04-28 08:56:24,735 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d6af57fc34a54a2eb4257a07aebfa0e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682672182360 2023-04-28 08:56:24,736 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 8924b3f618434331a5eabb968dc2e47a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682672183527 2023-04-28 08:56:24,736 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9b59721c3f2d4283bcd89b9eac5604ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682672183527 2023-04-28 08:56:24,747 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#596 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:24,753 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#595 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:24,768 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/41017c3eeb204159a389d7c281572a05 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/41017c3eeb204159a389d7c281572a05 2023-04-28 08:56:24,772 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into 41017c3eeb204159a389d7c281572a05(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:24,772 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:24,772 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=10, startTime=1682672184731; duration=0sec 2023-04-28 08:56:24,772 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:24,772 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-28 08:56:24,774 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82968 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-28 08:56:24,774 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/C is initiating minor compaction (all files) 2023-04-28 08:56:24,774 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/C in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,774 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52a7cfe9d68e49bbaa50937e234ca97f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ad457d54204e412a929b4911872d5eae, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/46ccd353888c48dd834d8ca4e42e9b16, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/47c02b2fe07740c882c79f9a3cdfdb7f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/671a74933ba847d0a2da3729785ac2b7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c8d562449caa49ee86e4bb991972cb99, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94b6aef839644e87bec8384552b5f365] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=81.0 K 2023-04-28 08:56:24,775 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 52a7cfe9d68e49bbaa50937e234ca97f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1682672179399 2023-04-28 08:56:24,775 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ad457d54204e412a929b4911872d5eae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1682672180095 2023-04-28 08:56:24,775 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 46ccd353888c48dd834d8ca4e42e9b16, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1682672180265 2023-04-28 08:56:24,776 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 47c02b2fe07740c882c79f9a3cdfdb7f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=299, earliestPutTs=1682672181073 2023-04-28 08:56:24,776 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 671a74933ba847d0a2da3729785ac2b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1682672181739 2023-04-28 08:56:24,776 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting c8d562449caa49ee86e4bb991972cb99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1682672182360 2023-04-28 08:56:24,776 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 94b6aef839644e87bec8384552b5f365, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682672183527 2023-04-28 08:56:24,786 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#C#compaction#597 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:24,798 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/ee97096e25914ade9671beeffd5c30d6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ee97096e25914ade9671beeffd5c30d6 2023-04-28 08:56:24,804 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/C of 92c5373160b2e33e00d0bc96891cd0cb into ee97096e25914ade9671beeffd5c30d6(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:24,804 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:24,804 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/C, priority=9, startTime=1682672184731; duration=0sec 2023-04-28 08:56:24,804 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:24,817 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:24,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:24,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:24,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:24,820 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:24,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:24,820 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:24,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,821 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,822 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,822 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,823 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:24,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:24,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:24,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:24,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:24,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:24,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:24,824 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:24,824 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,824 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:24,825 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:56:24,825 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:24,825 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:24,825 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:24,825 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:24,825 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:24,825 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:24,844 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428f2cda5574cff4f1999517f7c31dd2e15 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f2cda5574cff4f1999517f7c31dd2e15 2023-04-28 08:56:24,853 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=395, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/35c04107edc94903a36df5fcc2ba8bb7 2023-04-28 08:56:24,863 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/feb72acb408b49ac949af95cb2585755 2023-04-28 08:56:24,897 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=395 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/9f14ea4458914ca9932ae4d0be9d036a 2023-04-28 08:56:24,902 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/35c04107edc94903a36df5fcc2ba8bb7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/35c04107edc94903a36df5fcc2ba8bb7 2023-04-28 08:56:24,905 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/35c04107edc94903a36df5fcc2ba8bb7, entries=150, sequenceid=395, filesize=25.4 K 2023-04-28 08:56:24,906 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/feb72acb408b49ac949af95cb2585755 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/feb72acb408b49ac949af95cb2585755 2023-04-28 08:56:24,909 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/feb72acb408b49ac949af95cb2585755, entries=150, sequenceid=395, filesize=11.9 K 2023-04-28 08:56:24,910 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/9f14ea4458914ca9932ae4d0be9d036a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/9f14ea4458914ca9932ae4d0be9d036a 2023-04-28 08:56:24,913 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/9f14ea4458914ca9932ae4d0be9d036a, entries=150, sequenceid=395, filesize=11.9 K 2023-04-28 08:56:24,914 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 92c5373160b2e33e00d0bc96891cd0cb in 90ms, sequenceid=395, compaction requested=false 2023-04-28 08:56:24,914 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:24,914 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:24,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:24,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:24,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:24,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:24,916 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,916 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,916 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,916 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:24,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@247f1d15[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:24,918 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:24,918 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,920 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,920 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,920 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:24,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,920 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:24,920 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:24,920 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:24,926 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:24,926 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,926 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,926 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:24,929 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:24,929 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:24,929 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:24,929 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:24,929 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:24,931 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,931 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,931 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:24,931 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:24,933 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:24,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:24,935 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:24,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:24,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:24,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:24,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:24,935 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:24,935 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:24,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:24,935 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:24,936 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:24,936 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,936 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:24,936 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:25,036 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:25,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:25,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:25,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:25,059 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:25,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:25,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:25,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:25,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:25,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:25,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:25,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:25,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:25,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:25,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:25,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:25,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:25,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:25,063 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:25,063 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:25,063 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:25,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:25,067 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:25,067 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:25,067 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:25,068 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:56:25,068 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:25,069 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:25,069 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:25,069 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:25,069 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:25,069 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:25,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:25,073 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:25,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672185087, max timestamp=1682672185109Num uniques 859; 2023-04-28 08:56:25,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:56:25,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672185087, max timestamp=1682672185109Num uniques 859; 2023-04-28 08:56:25,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:56:25,111 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1682672185087, max timestamp=1682672185109Num uniques 859; 2023-04-28 08:56:25,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-28 08:56:25,141 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d74efa20e40947168ec4a418c8f9e86b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d74efa20e40947168ec4a418c8f9e86b 2023-04-28 08:56:25,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672245139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672245141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672245141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672245142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,210 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/274e7da8b24442a0aecb04ac8996c7e5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/274e7da8b24442a0aecb04ac8996c7e5 2023-04-28 08:56:25,214 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into 274e7da8b24442a0aecb04ac8996c7e5(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:25,214 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:25,214 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=10, startTime=1682672184731; duration=0sec 2023-04-28 08:56:25,215 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672245245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672245247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672245248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672245248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672245450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672245451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672245451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672245451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,563 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=404, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ae255786adc04c0783c0e580d656aca2 2023-04-28 08:56:25,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672245753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672245754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672245755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672245756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:25,976 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/bd2a53355f3e44b1ae5c6bd8a6a0521f 2023-04-28 08:56:25,985 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/6677afcb9699488ea4de20027738523f 2023-04-28 08:56:25,989 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ae255786adc04c0783c0e580d656aca2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ae255786adc04c0783c0e580d656aca2 2023-04-28 08:56:25,992 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ae255786adc04c0783c0e580d656aca2, entries=250, sequenceid=404, filesize=39.2 K 2023-04-28 08:56:25,993 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/bd2a53355f3e44b1ae5c6bd8a6a0521f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/bd2a53355f3e44b1ae5c6bd8a6a0521f 2023-04-28 08:56:25,997 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/bd2a53355f3e44b1ae5c6bd8a6a0521f, entries=150, sequenceid=404, filesize=11.9 K 2023-04-28 08:56:25,998 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/6677afcb9699488ea4de20027738523f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/6677afcb9699488ea4de20027738523f 2023-04-28 08:56:26,001 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/6677afcb9699488ea4de20027738523f, entries=150, sequenceid=404, filesize=11.9 K 2023-04-28 08:56:26,002 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 92c5373160b2e33e00d0bc96891cd0cb in 934ms, sequenceid=404, compaction requested=true 2023-04-28 08:56:26,002 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:26,002 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:26,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:26,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:26,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:26,002 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:26,004 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,004 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:26,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,005 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:26,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:26,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eb37bc[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:26,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:26,006 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,010 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:26,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:26,010 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:26,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,010 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:26,011 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:26,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:26,011 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:26,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:26,014 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:26,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:26,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:26,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:26,014 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:26,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:26,016 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:26,016 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,018 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,021 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:26,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,021 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:26,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:26,021 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:26,021 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:26,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:26,122 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:26,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:26,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:26,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:26,125 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:26,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:26,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:26,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:26,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,125 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,127 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:26,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:26,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,127 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:26,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:26,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:26,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:26,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:26,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:26,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:26,128 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:26,128 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:26,128 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:26,128 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-28 08:56:26,129 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:26,129 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:26,129 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:26,129 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:26,129 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:26,129 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:26,140 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428389995f42e7840e4af999b07e6ec7539 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428389995f42e7840e4af999b07e6ec7539 2023-04-28 08:56:26,148 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=442, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/05c1aeb7cf874558ada29435c3d0bd5c 2023-04-28 08:56:26,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:26,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:26,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672246263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672246263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672246266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672246267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672246367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672246368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672246370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,558 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d8e95291f8464e2f8179c0bc0217cd02 2023-04-28 08:56:26,569 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/52926550c3364b259cae7335c6ebdce0 2023-04-28 08:56:26,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672246570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672246572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,573 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/05c1aeb7cf874558ada29435c3d0bd5c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/05c1aeb7cf874558ada29435c3d0bd5c 2023-04-28 08:56:26,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672246573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,577 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/05c1aeb7cf874558ada29435c3d0bd5c, entries=150, sequenceid=442, filesize=25.4 K 2023-04-28 08:56:26,578 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d8e95291f8464e2f8179c0bc0217cd02 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d8e95291f8464e2f8179c0bc0217cd02 2023-04-28 08:56:26,581 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d8e95291f8464e2f8179c0bc0217cd02, entries=150, sequenceid=442, filesize=11.9 K 2023-04-28 08:56:26,581 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/52926550c3364b259cae7335c6ebdce0 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52926550c3364b259cae7335c6ebdce0 2023-04-28 08:56:26,584 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52926550c3364b259cae7335c6ebdce0, entries=150, sequenceid=442, filesize=11.9 K 2023-04-28 08:56:26,585 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 92c5373160b2e33e00d0bc96891cd0cb in 457ms, sequenceid=442, compaction requested=true 2023-04-28 08:56:26,585 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:26,585 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:26,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:26,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:26,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:26,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:26,587 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:26,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,587 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:26,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48299ce4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:26,590 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:26,590 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,591 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:26,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:26,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:26,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:26,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:26,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:26,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:26,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:26,594 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:26,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:26,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:26,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:26,594 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:26,597 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,597 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:26,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,597 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:26,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:26,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:26,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:26,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:26,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:26,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:26,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:26,601 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:26,601 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:26,602 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:26,602 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:26,602 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:26,602 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,702 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:26,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:26,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:26,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:26,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:26,705 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:26,705 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:26,705 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:26,705 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:26,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,706 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,707 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:26,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:26,708 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,708 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:26,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:26,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:26,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:26,709 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:26,709 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:26,709 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:26,709 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:26,709 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:26,709 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:26,709 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:26,710 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-28 08:56:26,710 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:26,710 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:26,710 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:26,710 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:26,710 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:26,710 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:26,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:26,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:26,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672186891, max timestamp=1682672186900Num uniques 859; 2023-04-28 08:56:26,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672186891, max timestamp=1682672186900Num uniques 859; 2023-04-28 08:56:26,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:26,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672186891, max timestamp=1682672186900Num uniques 859; 2023-04-28 08:56:26,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:26,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:26,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682672186911, max timestamp=1682672186926Num uniques 1018; 2023-04-28 08:56:26,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 08:56:26,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682672186911, max timestamp=1682672186926Num uniques 1018; 2023-04-28 08:56:26,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 08:56:26,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1682672186911, max timestamp=1682672186926Num uniques 1018; 2023-04-28 08:56:26,929 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-28 08:56:26,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:26,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672246957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:26,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672186943, max timestamp=1682672186962Num uniques 1077; 2023-04-28 08:56:26,964 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 08:56:26,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672186943, max timestamp=1682672186962Num uniques 1077; 2023-04-28 08:56:26,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 08:56:26,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1682672186943, max timestamp=1682672186962Num uniques 1077; 2023-04-28 08:56:26,965 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-28 08:56:27,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672247003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672247007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672247066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672247109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,113 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672247111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,128 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428296cfd30c20a43489e8248ebfeb22856 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428296cfd30c20a43489e8248ebfeb22856 2023-04-28 08:56:27,133 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=451, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/b5cd945b5c7b4d139b0e5edb38b657d1 2023-04-28 08:56:27,145 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/fa01e53cd5fc4d06a1af4d92de9fbbc6 2023-04-28 08:56:27,155 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/62d5b414e05045e298ce1b60ce78c242 2023-04-28 08:56:27,160 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/b5cd945b5c7b4d139b0e5edb38b657d1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/b5cd945b5c7b4d139b0e5edb38b657d1 2023-04-28 08:56:27,164 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/b5cd945b5c7b4d139b0e5edb38b657d1, entries=150, sequenceid=451, filesize=25.4 K 2023-04-28 08:56:27,164 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/fa01e53cd5fc4d06a1af4d92de9fbbc6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/fa01e53cd5fc4d06a1af4d92de9fbbc6 2023-04-28 08:56:27,168 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/fa01e53cd5fc4d06a1af4d92de9fbbc6, entries=150, sequenceid=451, filesize=11.9 K 2023-04-28 08:56:27,169 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/62d5b414e05045e298ce1b60ce78c242 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/62d5b414e05045e298ce1b60ce78c242 2023-04-28 08:56:27,172 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/62d5b414e05045e298ce1b60ce78c242, entries=150, sequenceid=451, filesize=11.9 K 2023-04-28 08:56:27,172 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 92c5373160b2e33e00d0bc96891cd0cb in 462ms, sequenceid=451, compaction requested=true 2023-04-28 08:56:27,173 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:27,173 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:27,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:27,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:27,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:27,175 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,175 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,176 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,176 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:27,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3e6ef3[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:27,177 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:27,177 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,178 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,178 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:27,179 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,179 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:27,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:27,179 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:27,180 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:27,180 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,180 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,180 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:27,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:27,183 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:27,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:27,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:27,183 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:27,183 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:27,184 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,184 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,184 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,186 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:27,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:27,189 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,189 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,190 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:27,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,190 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:27,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:27,271 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=147.60 KB heapSize=419.16 KB 2023-04-28 08:56:27,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:27,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:56:27,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:27,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:56:27,271 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:27,271 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-28 08:56:27,285 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428b688db8635044d78a4853dc78768c9fd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b688db8635044d78a4853dc78768c9fd 2023-04-28 08:56:27,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672247282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,289 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=523, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6 2023-04-28 08:56:27,290 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:27,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:27,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672247286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:27,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:27,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:27,294 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:27,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:27,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:27,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,295 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,296 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,296 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:27,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:27,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:27,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:27,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:27,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,297 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:27,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:27,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:27,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:27,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:27,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:27,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:27,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:27,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:27,304 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,304 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d297d5c124484865a9ae35763d15ba2a 2023-04-28 08:56:27,304 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,305 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f54bf4[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:27,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:27,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,307 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,307 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:27,307 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:27,307 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:27,308 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,308 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:27,313 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:27,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,313 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a84957c9ab9a4bc59f94dac1db173434 2023-04-28 08:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672247312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:27,316 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:27,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:27,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:27,316 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:27,316 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:27,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6 2023-04-28 08:56:27,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672247315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,320 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,322 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6, entries=200, sequenceid=523, filesize=32.3 K 2023-04-28 08:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,323 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/d297d5c124484865a9ae35763d15ba2a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d297d5c124484865a9ae35763d15ba2a 2023-04-28 08:56:27,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:27,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:27,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,324 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:27,324 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,325 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,324 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,325 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:27,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d297d5c124484865a9ae35763d15ba2a, entries=150, sequenceid=523, filesize=11.9 K 2023-04-28 08:56:27,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a84957c9ab9a4bc59f94dac1db173434 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a84957c9ab9a4bc59f94dac1db173434 2023-04-28 08:56:27,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a84957c9ab9a4bc59f94dac1db173434, entries=150, sequenceid=523, filesize=11.9 K 2023-04-28 08:56:27,330 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~419.11 KB/429168, currentSize=33.54 KB/34350 for 92c5373160b2e33e00d0bc96891cd0cb in 59ms, sequenceid=523, compaction requested=true 2023-04-28 08:56:27,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:27,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:27,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:27,331 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:56:27,331 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:56:27,331 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:27,332 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 178082 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:56:27,332 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:27,332 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73798 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:56:27,332 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,333 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:27,333 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/274e7da8b24442a0aecb04ac8996c7e5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/35c04107edc94903a36df5fcc2ba8bb7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ae255786adc04c0783c0e580d656aca2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/05c1aeb7cf874558ada29435c3d0bd5c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/b5cd945b5c7b4d139b0e5edb38b657d1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=173.9 K 2023-04-28 08:56:27,333 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,333 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/41017c3eeb204159a389d7c281572a05, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/feb72acb408b49ac949af95cb2585755, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/bd2a53355f3e44b1ae5c6bd8a6a0521f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d8e95291f8464e2f8179c0bc0217cd02, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/fa01e53cd5fc4d06a1af4d92de9fbbc6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d297d5c124484865a9ae35763d15ba2a] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=72.1 K 2023-04-28 08:56:27,333 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 274e7da8b24442a0aecb04ac8996c7e5, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682672183527 2023-04-28 08:56:27,333 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 41017c3eeb204159a389d7c281572a05, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682672183527 2023-04-28 08:56:27,333 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 35c04107edc94903a36df5fcc2ba8bb7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682672183887 2023-04-28 08:56:27,333 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting feb72acb408b49ac949af95cb2585755, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682672183887 2023-04-28 08:56:27,333 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ae255786adc04c0783c0e580d656aca2, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682672185035 2023-04-28 08:56:27,334 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bd2a53355f3e44b1ae5c6bd8a6a0521f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682672185064 2023-04-28 08:56:27,334 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 05c1aeb7cf874558ada29435c3d0bd5c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1682672185130 2023-04-28 08:56:27,334 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d8e95291f8464e2f8179c0bc0217cd02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1682672185130 2023-04-28 08:56:27,334 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting b5cd945b5c7b4d139b0e5edb38b657d1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682672186260 2023-04-28 08:56:27,334 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting fa01e53cd5fc4d06a1af4d92de9fbbc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682672186260 2023-04-28 08:56:27,335 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 303e8a9f3b5a4dd9a2ffde8a8c6b94f6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682672186996 2023-04-28 08:56:27,335 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d297d5c124484865a9ae35763d15ba2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682672187001 2023-04-28 08:56:27,344 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#613 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:27,378 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#614 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:27,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:27,402 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:56:27,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:27,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:27,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:27,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:27,403 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:27,403 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:27,423 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304280a0493fbad144787b876b1e52246e002 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280a0493fbad144787b876b1e52246e002 2023-04-28 08:56:27,425 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:27,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:27,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:27,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:27,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:27,435 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:27,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:27,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:27,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,437 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:27,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:27,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:27,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:27,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,439 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:27,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:27,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:27,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:27,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:27,440 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:27,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:27,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:27,440 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:27,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,442 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,442 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,442 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=534, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/f8d895e744ae4327a41c43687ebe3dd5 2023-04-28 08:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d7fc6a1[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:27,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:27,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,446 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:27,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:27,446 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:27,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:27,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:27,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,449 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:27,452 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:27,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:27,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:27,452 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:27,452 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:27,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,455 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,455 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/58c24cc9b6bd43649f8316cc3795f8f7 2023-04-28 08:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,457 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:27,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,460 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,460 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,461 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:27,461 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:27,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672247461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672187453, max timestamp=1682672187464Num uniques 859; 2023-04-28 08:56:27,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:27,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672187453, max timestamp=1682672187464Num uniques 859; 2023-04-28 08:56:27,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672187453, max timestamp=1682672187464Num uniques 859; 2023-04-28 08:56:27,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:27,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:27,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672247536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,561 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:27,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:27,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:27,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:27,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:27,567 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:27,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:27,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:27,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672247566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,568 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:27,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:27,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:27,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:27,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:27,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:27,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:27,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:27,574 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,574 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:27,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:27,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:27,575 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:27,576 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,576 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,579 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,579 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:27,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3935a356[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:27,581 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:27,581 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,583 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,583 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:27,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:27,583 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,584 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:27,584 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:27,585 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:27,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,585 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:27,588 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:27,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:27,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:27,588 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:27,588 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:27,590 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,590 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,590 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,590 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,592 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:27,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:27,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,595 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,595 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,595 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:27,595 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:27,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:27,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,594 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,595 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672247618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672247622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672247642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,695 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:27,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:27,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:27,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:27,701 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:27,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:27,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:27,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,701 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,704 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,704 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,704 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,704 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:27,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:27,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:27,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:27,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,705 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:27,706 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:27,706 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:27,706 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,706 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:27,706 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:27,706 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,706 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:27,706 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:27,706 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,706 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:27,706 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:27,707 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,707 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,708 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,708 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b21b75[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:27,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:27,711 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,712 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:27,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:27,712 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:27,712 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,713 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:27,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:27,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,715 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:27,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:27,717 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:27,717 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:27,717 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:27,717 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:27,717 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:27,720 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,720 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,720 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,721 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,722 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:27,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:27,736 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,736 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,736 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:27,736 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:27,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672247770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,781 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/0da60f5ae61a43788809accc2cb077cf as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0da60f5ae61a43788809accc2cb077cf 2023-04-28 08:56:27,785 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into 0da60f5ae61a43788809accc2cb077cf(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:27,786 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:27,786 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=10, startTime=1682672187331; duration=0sec 2023-04-28 08:56:27,786 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:27,786 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-28 08:56:27,788 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-28 08:56:27,788 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/C is initiating minor compaction (all files) 2023-04-28 08:56:27,788 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/C in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,788 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ee97096e25914ade9671beeffd5c30d6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/9f14ea4458914ca9932ae4d0be9d036a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/6677afcb9699488ea4de20027738523f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52926550c3364b259cae7335c6ebdce0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/62d5b414e05045e298ce1b60ce78c242, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a84957c9ab9a4bc59f94dac1db173434] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=72.0 K 2023-04-28 08:56:27,788 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ee97096e25914ade9671beeffd5c30d6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1682672183527 2023-04-28 08:56:27,789 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 9f14ea4458914ca9932ae4d0be9d036a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=395, earliestPutTs=1682672183887 2023-04-28 08:56:27,789 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 6677afcb9699488ea4de20027738523f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1682672185064 2023-04-28 08:56:27,789 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 52926550c3364b259cae7335c6ebdce0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1682672185130 2023-04-28 08:56:27,790 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 62d5b414e05045e298ce1b60ce78c242, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1682672186260 2023-04-28 08:56:27,790 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a84957c9ab9a4bc59f94dac1db173434, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682672187001 2023-04-28 08:56:27,801 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#C#compaction#618 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:27,809 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/bef65fd57110409dafe51e41577b809d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/bef65fd57110409dafe51e41577b809d 2023-04-28 08:56:27,813 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/C of 92c5373160b2e33e00d0bc96891cd0cb into bef65fd57110409dafe51e41577b809d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:27,813 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:27,813 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/C, priority=10, startTime=1682672187331; duration=0sec 2023-04-28 08:56:27,813 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:27,837 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:27,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:27,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:27,840 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:27,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:27,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:27,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,840 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,841 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,841 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,842 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,842 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:27,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:27,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:27,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:27,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:27,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:27,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:27,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:27,843 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,843 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:27,843 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:27,843 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,843 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:27,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:27,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:27,844 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:27,846 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,846 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,847 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,847 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cd3449c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:27,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:27,848 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:27,850 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672247847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:27,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:27,850 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:27,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,850 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:27,852 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:27,852 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,852 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,852 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:27,855 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:27,855 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:27,855 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:27,855 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:27,855 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:27,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,856 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,857 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:27,857 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:27,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,859 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,862 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:27,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:27,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:27,862 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,862 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,862 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:27,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:27,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:27,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:27,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/e116986570554efaa009fee214a51b49 2023-04-28 08:56:27,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/f8d895e744ae4327a41c43687ebe3dd5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/f8d895e744ae4327a41c43687ebe3dd5 2023-04-28 08:56:27,875 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/f8d895e744ae4327a41c43687ebe3dd5, entries=250, sequenceid=534, filesize=39.2 K 2023-04-28 08:56:27,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/58c24cc9b6bd43649f8316cc3795f8f7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/58c24cc9b6bd43649f8316cc3795f8f7 2023-04-28 08:56:27,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/58c24cc9b6bd43649f8316cc3795f8f7, entries=150, sequenceid=534, filesize=11.9 K 2023-04-28 08:56:27,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/e116986570554efaa009fee214a51b49 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/e116986570554efaa009fee214a51b49 2023-04-28 08:56:27,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/e116986570554efaa009fee214a51b49, entries=150, sequenceid=534, filesize=11.9 K 2023-04-28 08:56:27,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 92c5373160b2e33e00d0bc96891cd0cb in 482ms, sequenceid=534, compaction requested=false 2023-04-28 08:56:27,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:27,963 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:27,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:27,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:27,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:27,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:27,966 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:27,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:27,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:27,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,966 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,968 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:27,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:27,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:27,968 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:27,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:27,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:27,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:27,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:27,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:27,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:27,969 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:27,970 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:27,970 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:27,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:27,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:27,970 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-28 08:56:27,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:27,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:27,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:27,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:27,970 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:27,971 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:27,983 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428415be84d8c7e4ebcbf5ad07b14a1247e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428415be84d8c7e4ebcbf5ad07b14a1247e 2023-04-28 08:56:27,988 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=572, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/0a4d51edfea74a6b90493d9f869cba62 2023-04-28 08:56:27,998 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/ed5e9ab0b5d34bc092c91b60a164a706 2023-04-28 08:56:28,010 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/d9e30f025f634199801d35a8547fb3f9 2023-04-28 08:56:28,014 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/0a4d51edfea74a6b90493d9f869cba62 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/0a4d51edfea74a6b90493d9f869cba62 2023-04-28 08:56:28,017 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/0a4d51edfea74a6b90493d9f869cba62, entries=150, sequenceid=572, filesize=25.4 K 2023-04-28 08:56:28,018 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/ed5e9ab0b5d34bc092c91b60a164a706 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ed5e9ab0b5d34bc092c91b60a164a706 2023-04-28 08:56:28,021 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ed5e9ab0b5d34bc092c91b60a164a706, entries=150, sequenceid=572, filesize=11.9 K 2023-04-28 08:56:28,022 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/d9e30f025f634199801d35a8547fb3f9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/d9e30f025f634199801d35a8547fb3f9 2023-04-28 08:56:28,026 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/d9e30f025f634199801d35a8547fb3f9, entries=150, sequenceid=572, filesize=11.9 K 2023-04-28 08:56:28,026 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 92c5373160b2e33e00d0bc96891cd0cb in 56ms, sequenceid=572, compaction requested=true 2023-04-28 08:56:28,026 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:28,026 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:28,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:28,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:28,027 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:28,028 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,028 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,029 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,029 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fdac769[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:28,031 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:28,031 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,032 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,032 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:28,032 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:28,032 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:28,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:28,035 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,035 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,035 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,035 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:28,038 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:28,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:28,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:28,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:28,038 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:28,039 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,039 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,039 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,040 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,042 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,044 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:28,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:28,045 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:28,045 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:28,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,045 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:28,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,145 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:28,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:28,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:28,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:28,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:28,151 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:28,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:28,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:28,151 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:56:28,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:28,151 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:28,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:28,153 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,154 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:28,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:28,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:28,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:28,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:28,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,158 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:28,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:28,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:28,163 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,163 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:28,163 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:28,163 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:28,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:28,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:28,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:28,166 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,166 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,167 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,167 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:28,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b61f4d7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:28,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:28,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,170 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:28,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:28,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:28,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:28,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:28,172 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,172 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304281401f781505d4501833aa2eb430dafef to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281401f781505d4501833aa2eb430dafef 2023-04-28 08:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:28,176 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:28,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:28,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:28,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:28,176 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:28,178 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,179 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,179 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=582, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1 2023-04-28 08:56:28,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672188172, max timestamp=1682672188179Num uniques 859; 2023-04-28 08:56:28,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:56:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672188172, max timestamp=1682672188179Num uniques 859; 2023-04-28 08:56:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672188172, max timestamp=1682672188179Num uniques 859; 2023-04-28 08:56:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:56:28,181 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:56:28,182 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,184 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,190 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,191 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:28,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,191 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,191 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:28,191 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:28,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,207 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/9b59106d9cd3496f94e89d80495539a5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9b59106d9cd3496f94e89d80495539a5 2023-04-28 08:56:28,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672248205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672248206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672248207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672248207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,211 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into 9b59106d9cd3496f94e89d80495539a5(size=26.3 K), total size for store is 90.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:28,211 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:28,211 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=10, startTime=1682672187331; duration=0sec 2023-04-28 08:56:28,211 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:28,291 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:28,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:28,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:28,295 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:28,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:28,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:28,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,295 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,296 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,296 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:28,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:28,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:28,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:28,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:28,298 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:28,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,298 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:28,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:28,299 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:28,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:28,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:28,299 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:28,300 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:28,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c6f337d[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:28,303 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:28,303 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,306 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:28,306 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:28,306 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:28,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,306 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:28,307 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,307 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:28,307 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,307 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:28,310 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:28,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:28,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:28,310 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:28,310 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:28,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672248311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,314 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672248312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672248312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672248312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,317 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,320 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:28,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:28,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:28,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,320 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,320 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:28,320 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:28,421 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:28,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:28,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:28,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:28,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:28,424 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:28,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:28,424 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:28,425 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,425 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,427 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,428 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,428 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:28,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:28,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:28,429 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,429 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:28,429 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:28,429 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:28,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:28,431 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,432 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a3e71c8[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:28,433 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:28,433 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,434 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:28,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:28,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:28,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,435 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:28,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:28,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,436 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:28,440 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:28,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:28,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:28,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:28,440 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:28,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,444 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,445 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,448 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:28,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:28,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:28,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,449 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:28,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,449 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:28,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59128 deadline: 1682672248471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,475 DEBUG [Thread-4007] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., hostname=jenkins-hbase6.apache.org,40909,1682672018517, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-28 08:56:28,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672248515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672248517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672248517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.20:59060 deadline: 1682672248517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,549 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:28,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:28,553 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:28,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:28,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:28,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:28,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:28,556 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:28,556 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,556 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:28,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:28,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:28,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:28,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:28,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:28,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:28,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:28,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:28,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,559 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3761a69[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:28,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:28,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,563 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:28,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:28,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:28,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:28,565 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:28,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:28,568 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:28,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:28,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:28,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:28,568 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:28,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:28,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,570 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:28,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,571 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,573 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:28,574 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,574 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:28,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:28,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/641e0fa5b97d491a91d076eea6470a2e 2023-04-28 08:56:28,608 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/94ee2029008f4893b00477860f5d4284 2023-04-28 08:56:28,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1 2023-04-28 08:56:28,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1, entries=350, sequenceid=582, filesize=52.9 K 2023-04-28 08:56:28,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/641e0fa5b97d491a91d076eea6470a2e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/641e0fa5b97d491a91d076eea6470a2e 2023-04-28 08:56:28,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/641e0fa5b97d491a91d076eea6470a2e, entries=150, sequenceid=582, filesize=11.9 K 2023-04-28 08:56:28,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/94ee2029008f4893b00477860f5d4284 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94ee2029008f4893b00477860f5d4284 2023-04-28 08:56:28,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94ee2029008f4893b00477860f5d4284, entries=150, sequenceid=582, filesize=11.9 K 2023-04-28 08:56:28,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 92c5373160b2e33e00d0bc96891cd0cb in 478ms, sequenceid=582, compaction requested=true 2023-04-28 08:56:28,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:28,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:28,629 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:56:28,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:28,629 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:56:28,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:28,631 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 147322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:56:28,631 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:28,631 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,631 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9b59106d9cd3496f94e89d80495539a5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/f8d895e744ae4327a41c43687ebe3dd5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/0a4d51edfea74a6b90493d9f869cba62, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=143.9 K 2023-04-28 08:56:28,631 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:56:28,631 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:28,631 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,631 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9b59106d9cd3496f94e89d80495539a5, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682672187001 2023-04-28 08:56:28,632 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0da60f5ae61a43788809accc2cb077cf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/58c24cc9b6bd43649f8316cc3795f8f7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ed5e9ab0b5d34bc092c91b60a164a706, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/641e0fa5b97d491a91d076eea6470a2e] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=48.5 K 2023-04-28 08:56:28,632 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting f8d895e744ae4327a41c43687ebe3dd5, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672187280 2023-04-28 08:56:28,632 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 0da60f5ae61a43788809accc2cb077cf, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682672187001 2023-04-28 08:56:28,632 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 0a4d51edfea74a6b90493d9f869cba62, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1682672187504 2023-04-28 08:56:28,633 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 58c24cc9b6bd43649f8316cc3795f8f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672187390 2023-04-28 08:56:28,633 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 9a2b5c0d6c924a3da87c34c2cc7f8ff1, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672188090 2023-04-28 08:56:28,633 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting ed5e9ab0b5d34bc092c91b60a164a706, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1682672187504 2023-04-28 08:56:28,634 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 641e0fa5b97d491a91d076eea6470a2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672188141 2023-04-28 08:56:28,644 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#626 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:28,645 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#625 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:28,656 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/825dd9f8929b44238c882afc32527224 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/825dd9f8929b44238c882afc32527224 2023-04-28 08:56:28,661 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into 825dd9f8929b44238c882afc32527224(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:28,661 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:28,661 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=12, startTime=1682672188629; duration=0sec 2023-04-28 08:56:28,661 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:28,661 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:56:28,662 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:56:28,662 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/C is initiating minor compaction (all files) 2023-04-28 08:56:28,662 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/C in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,663 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/bef65fd57110409dafe51e41577b809d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/e116986570554efaa009fee214a51b49, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/d9e30f025f634199801d35a8547fb3f9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94ee2029008f4893b00477860f5d4284] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=48.4 K 2023-04-28 08:56:28,663 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting bef65fd57110409dafe51e41577b809d, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1682672187001 2023-04-28 08:56:28,663 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting e116986570554efaa009fee214a51b49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=534, earliestPutTs=1682672187390 2023-04-28 08:56:28,664 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting d9e30f025f634199801d35a8547fb3f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1682672187504 2023-04-28 08:56:28,664 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 94ee2029008f4893b00477860f5d4284, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672188141 2023-04-28 08:56:28,668 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/38a4f7cca70849fb94ac5268f7a769a2 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/38a4f7cca70849fb94ac5268f7a769a2 2023-04-28 08:56:28,672 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#C#compaction#627 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:28,672 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into 38a4f7cca70849fb94ac5268f7a769a2(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:28,672 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:28,672 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=12, startTime=1682672188629; duration=0sec 2023-04-28 08:56:28,673 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:28,674 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:28,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:28,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:28,678 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:28,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:28,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:28,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,684 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:28,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:28,684 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,685 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,685 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:28,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:28,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:28,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:28,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:28,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:28,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:28,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:28,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:28,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:28,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:28,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:28,686 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-28 08:56:28,686 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:28,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:28,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:28,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:28,687 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/8ca3eff3b8554c4282a01a3c2fe1a75f as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8ca3eff3b8554c4282a01a3c2fe1a75f 2023-04-28 08:56:28,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:28,687 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:28,692 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/C of 92c5373160b2e33e00d0bc96891cd0cb into 8ca3eff3b8554c4282a01a3c2fe1a75f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:28,692 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:28,692 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/C, priority=12, startTime=1682672188630; duration=0sec 2023-04-28 08:56:28,693 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:28,706 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428066eab1e98e34c349550828586d91a41 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428066eab1e98e34c349550828586d91a41 2023-04-28 08:56:28,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:28,821 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:28,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672248823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672248823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672248824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672248824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672248927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672248927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672248927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:28,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:28,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672248927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,111 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=622, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/55fc0c0dee1f4276808a812fcf2a775b 2023-04-28 08:56:29,121 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/a0964647089c4d3abf0f091afe3d9460 2023-04-28 08:56:29,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672249128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672249131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672249131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672249134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,137 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/2e72ec29a75f4da089d9b0dfa4c7ff1c 2023-04-28 08:56:29,140 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/55fc0c0dee1f4276808a812fcf2a775b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/55fc0c0dee1f4276808a812fcf2a775b 2023-04-28 08:56:29,144 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/55fc0c0dee1f4276808a812fcf2a775b, entries=150, sequenceid=622, filesize=25.4 K 2023-04-28 08:56:29,145 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/a0964647089c4d3abf0f091afe3d9460 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a0964647089c4d3abf0f091afe3d9460 2023-04-28 08:56:29,148 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a0964647089c4d3abf0f091afe3d9460, entries=150, sequenceid=622, filesize=11.9 K 2023-04-28 08:56:29,149 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/2e72ec29a75f4da089d9b0dfa4c7ff1c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/2e72ec29a75f4da089d9b0dfa4c7ff1c 2023-04-28 08:56:29,156 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/2e72ec29a75f4da089d9b0dfa4c7ff1c, entries=150, sequenceid=622, filesize=11.9 K 2023-04-28 08:56:29,156 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 92c5373160b2e33e00d0bc96891cd0cb in 470ms, sequenceid=622, compaction requested=false 2023-04-28 08:56:29,157 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:29,157 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:29,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:29,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:29,157 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:29,158 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,159 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,159 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,159 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:29,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6755819c[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:29,161 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:29,161 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,162 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:29,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:29,163 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:29,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,163 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:29,165 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,165 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:29,165 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,165 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:29,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:29,168 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:29,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:29,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:29,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:29,168 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:29,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,170 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,171 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:29,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,174 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:29,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:29,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,174 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,174 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:29,174 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:29,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,274 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:29,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:29,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:29,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:29,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:29,278 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:29,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:29,278 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:29,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,279 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,280 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,281 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:29,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:29,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:29,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:29,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:29,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:29,282 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:29,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:29,282 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-28 08:56:29,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:29,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:29,282 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:29,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:29,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:29,283 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:29,295 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428d09a98ea3534429f86cba02855390330 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d09a98ea3534429f86cba02855390330 2023-04-28 08:56:29,301 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=631, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ff4bd8f281ff480fa2903b56c8d9c5d3 2023-04-28 08:56:29,345 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/b2346bc2852f44b682c6d06df3cc5ad4 2023-04-28 08:56:29,360 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/590a2073ab8c4ee7b439ae7738fd6758 2023-04-28 08:56:29,365 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/ff4bd8f281ff480fa2903b56c8d9c5d3 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ff4bd8f281ff480fa2903b56c8d9c5d3 2023-04-28 08:56:29,368 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ff4bd8f281ff480fa2903b56c8d9c5d3, entries=150, sequenceid=631, filesize=25.4 K 2023-04-28 08:56:29,369 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/b2346bc2852f44b682c6d06df3cc5ad4 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/b2346bc2852f44b682c6d06df3cc5ad4 2023-04-28 08:56:29,372 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/b2346bc2852f44b682c6d06df3cc5ad4, entries=150, sequenceid=631, filesize=11.9 K 2023-04-28 08:56:29,372 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/590a2073ab8c4ee7b439ae7738fd6758 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/590a2073ab8c4ee7b439ae7738fd6758 2023-04-28 08:56:29,377 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/590a2073ab8c4ee7b439ae7738fd6758, entries=150, sequenceid=631, filesize=11.9 K 2023-04-28 08:56:29,378 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 92c5373160b2e33e00d0bc96891cd0cb in 96ms, sequenceid=631, compaction requested=true 2023-04-28 08:56:29,378 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:29,378 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:29,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:29,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:29,378 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:29,384 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,384 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,385 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1254a2d0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:29,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:29,387 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,389 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,389 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:29,389 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:29,389 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:29,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,389 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:29,390 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:29,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,390 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,391 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:29,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:29,394 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:29,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:29,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:29,394 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:29,394 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:29,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,396 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,396 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,397 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,403 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:29,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:29,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:29,403 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,403 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,403 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:29,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:29,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-28 08:56:29,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:29,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:29,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:29,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:29,467 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:29,467 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:29,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672249493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672249493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672249493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672249494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,504 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:29,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:29,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:29,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:29,507 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:29,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:29,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:29,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,507 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,508 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,509 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,509 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,509 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:29,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:29,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:29,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:29,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:29,510 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:29,510 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,510 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:29,511 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:29,511 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:29,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:29,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:29,511 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:29,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,512 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,512 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,513 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3d3163[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:29,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:29,515 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,516 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,516 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:29,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:29,517 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:29,517 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,517 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:29,522 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,523 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:29,523 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,523 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:29,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:29,526 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:29,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:29,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:29,526 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:29,526 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:29,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,529 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,530 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,531 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,534 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,534 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,535 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:29,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:29,535 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,536 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:29,536 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:29,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,536 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672249598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672249598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672249598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672249599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,636 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:29,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:29,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:29,640 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:29,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:29,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:29,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,640 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,653 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,653 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,654 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:29,655 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:29,655 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,655 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:29,655 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:29,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:29,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:29,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:29,656 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:29,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:29,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:29,656 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:29,661 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,661 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,662 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78abcf3f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:29,663 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:29,663 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,667 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:29,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:29,667 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:29,667 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,667 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:29,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:29,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,669 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:29,669 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:29,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:29,671 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:29,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:29,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:29,671 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:29,672 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:29,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,673 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,674 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,675 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,680 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,679 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:29,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:29,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,680 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:29,680 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:29,680 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:29,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,780 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:29,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:29,784 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:29,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:29,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:29,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,784 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,786 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,786 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:29,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:29,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:29,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:29,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,787 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:29,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:29,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:29,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:29,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:29,788 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:29,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:29,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:29,788 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:29,790 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,790 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,791 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,791 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3339f8d7[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:29,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:29,792 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,794 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,794 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:29,794 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:29,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,794 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:29,794 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:29,796 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,796 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:29,796 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,796 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:29,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:29,799 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:29,799 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:29,799 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:29,799 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:29,799 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:29,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,801 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,802 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672249800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,802 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672249800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672249802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:29,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672249802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:29,807 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:29,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:29,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,807 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,808 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,808 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:29,808 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:29,883 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428b346e541aeef4e4694c298991c1450cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b346e541aeef4e4694c298991c1450cb 2023-04-28 08:56:29,909 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:29,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:29,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:29,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:29,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:29,911 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:29,911 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,911 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:29,911 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:29,912 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,912 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,913 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,913 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,913 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,914 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:29,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:29,914 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:29,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:29,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:29,915 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:29,915 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:29,917 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,917 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,918 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44c8e42[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:29,919 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:29,919 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,920 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,921 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:29,921 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:29,921 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:29,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,921 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:29,922 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:29,922 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,922 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,922 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:29,925 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:29,925 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:29,925 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:29,925 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:29,925 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:29,927 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:29,927 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,927 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:29,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:29,928 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,930 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:29,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,933 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:29,933 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,933 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:29,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:29,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:29,933 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:29,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:29,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:29,933 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,934 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:29,934 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:29,934 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:29,934 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,934 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:29,934 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:29,934 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:29,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,034 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:30,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:30,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:30,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:30,038 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:30,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:30,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:30,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,038 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,039 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,040 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,040 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,040 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:30,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:30,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:30,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:30,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:30,041 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:30,041 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,042 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:30,042 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:30,042 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,042 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:30,042 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:30,042 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,042 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:30,042 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:30,044 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,044 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,044 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,044 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26580181[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:30,045 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:30,046 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,047 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:30,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:30,047 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:30,047 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,047 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:30,051 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,051 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,052 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:30,054 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:30,054 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:30,054 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:30,054 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:30,054 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:30,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,056 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,057 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,059 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,061 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:30,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:30,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,061 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,062 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,062 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:30,062 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672250105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672250105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672250107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672250107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,162 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:30,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:30,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:30,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:30,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:30,165 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:30,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:30,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:30,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,168 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,168 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,169 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,169 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:30,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:30,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:30,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:30,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:30,170 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:30,170 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:30,171 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:30,172 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,173 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,173 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ace042[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:30,174 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:30,174 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,176 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:30,181 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:30,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:30,181 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:30,182 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:30,182 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,182 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,182 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:30,185 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:30,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:30,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:30,185 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:30,185 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:30,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,188 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,189 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,190 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,192 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:30,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:30,192 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,193 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,193 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,193 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,193 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:30,193 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:30,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:30,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,288 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=643, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/a14ddf8f3d134283a69036b957c65adc 2023-04-28 08:56:30,293 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:30,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:30,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:30,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:30,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:30,297 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:30,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:30,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:30,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,297 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,298 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,298 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,299 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:30,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:30,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:30,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:30,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:30,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:30,300 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,300 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:30,300 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:30,300 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,300 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:30,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:30,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:30,301 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:30,302 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,302 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,303 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c2f5fc0[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:30,304 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:30,304 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,306 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:30,306 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:30,306 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:30,307 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,307 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:30,308 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,308 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:30,308 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,308 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:30,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:30,311 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:30,311 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:30,311 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:30,311 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:30,311 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:30,313 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,313 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,313 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,314 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,315 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,318 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,318 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,318 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:30,318 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,318 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,318 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:30,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,319 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,319 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:30,319 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:30,419 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:30,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:30,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:30,423 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:30,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:30,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:30,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,423 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,426 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,426 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,427 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:30,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:30,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:30,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:30,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:30,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:30,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:30,428 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:30,428 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,428 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:30,428 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:30,429 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:30,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:30,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:30,429 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:30,430 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,430 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,431 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:30,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cf1f099[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:30,432 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:30,432 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,433 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:30,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:30,434 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:30,434 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,434 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:30,435 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:30,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:30,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:30,438 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:30,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:30,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:30,438 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:30,438 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:30,441 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,441 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,441 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,443 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:30,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:30,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,449 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:30,449 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:30,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:30,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,549 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:30,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:30,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:30,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:30,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:30,553 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:30,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:30,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:30,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,553 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,555 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,555 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:30,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:30,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:30,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:30,556 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:30,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:30,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:30,557 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:30,557 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:30,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,559 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,559 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:30,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a43be8f[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:30,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:30,561 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,563 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:30,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:30,563 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:30,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,563 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:30,564 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:30,564 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,565 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:30,568 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:30,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:30,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:30,568 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:30,568 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:30,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,570 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,572 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:30,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,575 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:30,575 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:30,574 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,574 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,575 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:30,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672250610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672250611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672250612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:30,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672250612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,675 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:30,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:30,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:30,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:30,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:30,678 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:30,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:30,678 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:30,679 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,679 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,681 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,681 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,682 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:30,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:30,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:30,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:30,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:30,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:30,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:30,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:30,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:30,683 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,683 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:30,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:30,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:30,684 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:30,685 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,685 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,686 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:30,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8aef75[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:30,687 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:30,687 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,688 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:30,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:30,688 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:30,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,688 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:30,690 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:30,690 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,690 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,690 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:30,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:30,692 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:30,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:30,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:30,692 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:30,692 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:30,695 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,695 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:30,695 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:30,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,696 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,697 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,699 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:30,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:30,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:30,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:30,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,700 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,700 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,700 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,700 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:30,700 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:30,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/2b6298010513465c8650dc57636e71d7 2023-04-28 08:56:30,699 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:30,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,700 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,701 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:30,701 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:30,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/8db62115d8f34cca85d8bd748c461408 2023-04-28 08:56:30,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/a14ddf8f3d134283a69036b957c65adc as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a14ddf8f3d134283a69036b957c65adc 2023-04-28 08:56:30,716 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a14ddf8f3d134283a69036b957c65adc, entries=450, sequenceid=643, filesize=66.7 K 2023-04-28 08:56:30,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/2b6298010513465c8650dc57636e71d7 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b6298010513465c8650dc57636e71d7 2023-04-28 08:56:30,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b6298010513465c8650dc57636e71d7, entries=150, sequenceid=643, filesize=11.9 K 2023-04-28 08:56:30,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/8db62115d8f34cca85d8bd748c461408 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8db62115d8f34cca85d8bd748c461408 2023-04-28 08:56:30,724 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8db62115d8f34cca85d8bd748c461408, entries=150, sequenceid=643, filesize=11.9 K 2023-04-28 08:56:30,725 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 92c5373160b2e33e00d0bc96891cd0cb in 1260ms, sequenceid=643, compaction requested=true 2023-04-28 08:56:30,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:30,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:30,725 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:56:30,725 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:56:30,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:30,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:30,727 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 147454 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:56:30,727 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:30,727 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:56:30,727 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,727 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/38a4f7cca70849fb94ac5268f7a769a2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/55fc0c0dee1f4276808a812fcf2a775b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ff4bd8f281ff480fa2903b56c8d9c5d3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a14ddf8f3d134283a69036b957c65adc] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=144.0 K 2023-04-28 08:56:30,727 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:30,727 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,727 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/825dd9f8929b44238c882afc32527224, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a0964647089c4d3abf0f091afe3d9460, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/b2346bc2852f44b682c6d06df3cc5ad4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b6298010513465c8650dc57636e71d7] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=48.6 K 2023-04-28 08:56:30,728 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 38a4f7cca70849fb94ac5268f7a769a2, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672188141 2023-04-28 08:56:30,728 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 825dd9f8929b44238c882afc32527224, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672188141 2023-04-28 08:56:30,728 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 55fc0c0dee1f4276808a812fcf2a775b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682672188197 2023-04-28 08:56:30,728 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a0964647089c4d3abf0f091afe3d9460, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682672188197 2023-04-28 08:56:30,728 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting ff4bd8f281ff480fa2903b56c8d9c5d3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1682672188822 2023-04-28 08:56:30,728 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting b2346bc2852f44b682c6d06df3cc5ad4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1682672188822 2023-04-28 08:56:30,729 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting a14ddf8f3d134283a69036b957c65adc, keycount=450, bloomtype=ROW, size=66.7 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1682672189453 2023-04-28 08:56:30,729 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2b6298010513465c8650dc57636e71d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1682672189459 2023-04-28 08:56:30,736 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:30,739 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#637 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:30,748 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/5cad09c717ac4dd38c6d1ac35898fef6 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5cad09c717ac4dd38c6d1ac35898fef6 2023-04-28 08:56:30,751 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into 5cad09c717ac4dd38c6d1ac35898fef6(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:30,752 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:30,752 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=12, startTime=1682672190725; duration=0sec 2023-04-28 08:56:30,752 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:30,752 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-28 08:56:30,753 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-28 08:56:30,753 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/C is initiating minor compaction (all files) 2023-04-28 08:56:30,753 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/C in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,753 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8ca3eff3b8554c4282a01a3c2fe1a75f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/2e72ec29a75f4da089d9b0dfa4c7ff1c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/590a2073ab8c4ee7b439ae7738fd6758, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8db62115d8f34cca85d8bd748c461408] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=48.6 K 2023-04-28 08:56:30,753 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8ca3eff3b8554c4282a01a3c2fe1a75f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1682672188141 2023-04-28 08:56:30,753 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 2e72ec29a75f4da089d9b0dfa4c7ff1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1682672188197 2023-04-28 08:56:30,754 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 590a2073ab8c4ee7b439ae7738fd6758, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1682672188822 2023-04-28 08:56:30,754 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 8db62115d8f34cca85d8bd748c461408, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1682672189459 2023-04-28 08:56:30,759 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#C#compaction#639 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:30,771 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/259a161c56284054a630d9303b0ef14b as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/259a161c56284054a630d9303b0ef14b 2023-04-28 08:56:30,774 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/C of 92c5373160b2e33e00d0bc96891cd0cb into 259a161c56284054a630d9303b0ef14b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:30,774 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:30,774 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/C, priority=12, startTime=1682672190726; duration=0sec 2023-04-28 08:56:30,774 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:30,800 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:30,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:30,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:30,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:30,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:30,803 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:30,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:30,803 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:30,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,804 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,806 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:30,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:30,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:30,806 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:30,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:30,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:30,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:30,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:30,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:30,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:30,807 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:30,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:30,808 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:30,808 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:30,808 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:30,833 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428873482e652034ae897ccce013e60763c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428873482e652034ae897ccce013e60763c 2023-04-28 08:56:30,839 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=665, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/7aa20d79321f40dba779421b6fd550cd 2023-04-28 08:56:31,250 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/05602445be444a81a353d70df70b4d1e 2023-04-28 08:56:31,260 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=665 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a9329d0102b1462091680a26ccf999ca 2023-04-28 08:56:31,263 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/7aa20d79321f40dba779421b6fd550cd as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/7aa20d79321f40dba779421b6fd550cd 2023-04-28 08:56:31,266 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/7aa20d79321f40dba779421b6fd550cd, entries=150, sequenceid=665, filesize=25.4 K 2023-04-28 08:56:31,267 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/05602445be444a81a353d70df70b4d1e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/05602445be444a81a353d70df70b4d1e 2023-04-28 08:56:31,270 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/05602445be444a81a353d70df70b4d1e, entries=150, sequenceid=665, filesize=11.9 K 2023-04-28 08:56:31,271 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/a9329d0102b1462091680a26ccf999ca as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a9329d0102b1462091680a26ccf999ca 2023-04-28 08:56:31,275 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a9329d0102b1462091680a26ccf999ca, entries=150, sequenceid=665, filesize=11.9 K 2023-04-28 08:56:31,276 INFO [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 92c5373160b2e33e00d0bc96891cd0cb in 468ms, sequenceid=665, compaction requested=false 2023-04-28 08:56:31,276 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:31,276 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:31,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:31,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:31,276 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:31,278 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,278 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,278 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,278 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b5d625b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:31,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:31,280 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,283 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:31,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:31,283 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:31,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,283 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:31,286 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:31,287 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,287 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,287 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:31,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:31,289 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:31,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:31,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:31,290 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:31,290 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:31,291 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,291 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,291 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,292 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,294 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:31,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:31,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:31,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,305 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,305 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,306 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:31,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,306 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:31,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,306 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,406 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:31,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:31,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:31,410 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:31,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:31,410 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:31,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,411 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,412 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,412 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,413 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,413 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:31,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:31,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:31,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:31,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:31,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:31,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:31,435 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:31,435 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,436 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:31,436 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:31,436 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:31,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:31,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:31,436 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:31,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,438 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,438 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:31,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83b2c91[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:31,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:31,440 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,441 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:31,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:31,441 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:31,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,442 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:31,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:31,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,444 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:31,444 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:31,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:31,447 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:31,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:31,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:31,447 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:31,447 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:31,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,448 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,449 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,449 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,451 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,453 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:31,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:31,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,454 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,453 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,454 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:31,454 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:31,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:31,554 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:31,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:31,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:31,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:31,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:31,567 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:31,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:31,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:31,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,567 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,569 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,569 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:31,572 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/1f23d2651ac849a1a1e2e73fbfed386e as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f23d2651ac849a1a1e2e73fbfed386e 2023-04-28 08:56:31,577 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into 1f23d2651ac849a1a1e2e73fbfed386e(size=26.6 K), total size for store is 52.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:31,577 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:31,577 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=12, startTime=1682672190725; duration=0sec 2023-04-28 08:56:31,577 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:31,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:31,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:31,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:31,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:31,591 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:31,591 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,591 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:31,592 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:31,592 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:31,592 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:31,594 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,594 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,595 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:31,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6feb5cdf[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:31,596 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:31,596 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,598 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,598 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:31,598 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:31,598 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:31,598 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,599 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:31,600 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:31,600 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,600 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,601 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:31,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:31,604 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:31,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:31,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:31,604 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:31,604 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:31,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,605 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,606 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,606 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,607 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,610 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,611 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,610 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,611 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:31,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:31,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:31,611 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:31,611 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:31,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:31,648 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-28 08:56:31,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:31,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:31,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:31,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:31,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:31,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:31,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1682672191666, max timestamp=1682672191676Num uniques 859; 2023-04-28 08:56:31,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-28 08:56:31,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672191666, max timestamp=1682672191676Num uniques 859; 2023-04-28 08:56:31,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:31,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1682672191666, max timestamp=1682672191676Num uniques 859; 2023-04-28 08:56:31,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase6:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-28 08:56:31,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672251705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672251705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672251706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,711 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:31,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:31,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672251709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:31,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:31,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:31,723 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:31,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:31,723 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:31,724 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,724 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,725 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,726 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,727 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,727 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:31,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:31,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:31,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:31,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,743 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:31,745 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:31,746 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:31,746 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,746 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:31,747 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:31,747 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:31,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:31,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:31,747 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:31,749 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,749 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,750 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:31,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2550e710[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:31,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:31,752 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,753 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:31,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:31,753 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:31,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:31,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,756 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:31,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,756 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:31,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:31,757 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:31,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:31,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:31,757 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:31,757 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:31,758 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,758 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,759 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,761 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,763 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,766 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,766 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,767 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,767 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:31,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:31,768 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:31,768 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:31,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672251811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672251811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672251812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:31,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672251814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,868 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:31,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:31,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:31,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:31,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:31,872 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:31,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:31,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:31,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,872 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,877 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,878 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:31,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:31,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:31,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:31,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:31,883 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:31,883 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:31,884 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:31,885 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,885 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,886 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,886 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ef165b[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:31,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:31,888 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,889 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:31,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:31,890 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:31,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,890 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:31,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:31,892 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,892 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:31,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:31,896 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:31,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:31,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:31,896 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:31,896 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:31,899 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,900 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:31,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:31,901 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:31,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,903 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:31,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:31,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:31,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:31,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:31,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:31,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:31,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:31,906 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:31,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:31,906 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:31,907 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:31,907 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:31,907 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:31,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:32,007 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:32,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:32,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:32,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:32,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:32,011 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:32,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:32,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:32,011 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:32,012 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,012 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,014 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:32,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:32,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,014 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,015 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:32,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40909] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672252013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:32,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:32,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:32,016 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:32,016 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:32,016 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:32,016 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:32,016 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:32,016 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:32,016 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:32,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672252014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672252014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,017 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:32,017 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:32,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:32,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:32,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:32,017 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:32,019 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:32,019 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:32,020 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,020 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40909] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672252018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:32,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:32,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@461f7d3[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:32,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:32,022 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,024 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,024 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,024 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:32,024 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:32,024 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:32,024 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:32,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:32,026 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:32,026 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:32,029 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:32,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:32,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:32,029 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:32,029 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:32,031 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,031 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:32,031 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:32,032 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,033 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,036 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:32,036 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:32,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:32,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:32,037 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:32,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:32,037 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:32,037 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:32,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-28 08:56:32,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:32,091 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042853d686f458e140079a249c8f30609c82 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042853d686f458e140079a249c8f30609c82 2023-04-28 08:56:32,097 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=678, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/32c2f7603686453f8de7d19507e7b4eb 2023-04-28 08:56:32,137 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:32,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:32,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(933): Client=jenkins//172.31.2.20 procedure request for: flush-table-proc 2023-04-28 08:56:32,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-28 08:56:32,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-28 08:56:32,141 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-28 08:56:32,141 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:32,141 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-28 08:56:32,141 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-28 08:56:32,142 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,142 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,144 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:32,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:32,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,144 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-28 08:56:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-28 08:56:32,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-28 08:56:32,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-28 08:56:32,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-28 08:56:32,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-28 08:56:32,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-28 08:56:32,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-28 08:56:32,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-28 08:56:32,146 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-28 08:56:32,147 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:32,147 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. started... 2023-04-28 08:56:32,147 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. as already flushing 2023-04-28 08:56:32,147 DEBUG [rs(jenkins-hbase6.apache.org,40909,1682672018517)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:32,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-28 08:56:32,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-28 08:56:32,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-28 08:56:32,147 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-28 08:56:32,149 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,149 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:32,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,150 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-28 08:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase6.apache.org,40909,1682672018517' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-28 08:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31595e24[Count = 0] remaining members to acquire global barrier 2023-04-28 08:56:32,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-28 08:56:32,152 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,153 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-28 08:56:32,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-28 08:56:32,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,153 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-28 08:56:32,153 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase6.apache.org,40909,1682672018517' in zk 2023-04-28 08:56:32,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-28 08:56:32,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:32,156 DEBUG [member: 'jenkins-hbase6.apache.org,40909,1682672018517' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-28 08:56:32,156 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:32,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:32,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase6.apache.org,40909,1682672018517': 2023-04-28 08:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase6.apache.org,40909,1682672018517' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-28 08:56:32,159 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-28 08:56:32,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-28 08:56:32,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-28 08:56:32,159 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-28 08:56:32,159 INFO [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-28 08:56:32,162 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,162 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:32,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,162 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-28 08:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-28 08:56:32,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-28 08:56:32,164 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-28 08:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,166 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-28 08:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-28 08:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-28 08:56:32,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-28 08:56:32,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-28 08:56:32,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-28 08:56:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-28 08:56:32,169 DEBUG [Thread-4011] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-28 08:56:32,169 DEBUG [Thread-4011] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-28 08:56:32,169 DEBUG [(jenkins-hbase6.apache.org,32801,1682672017275)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-28 08:56:32,169 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-28 08:56:32,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-28 08:56:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-28 08:56:32,227 DEBUG [Thread-4020] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32bf16cd to 127.0.0.1:54655 2023-04-28 08:56:32,227 DEBUG [Thread-4020] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:32,230 DEBUG [Thread-4016] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b59bdcf to 127.0.0.1:54655 2023-04-28 08:56:32,230 DEBUG [Thread-4012] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:54655 2023-04-28 08:56:32,230 DEBUG [Thread-4016] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:32,230 DEBUG [Thread-4012] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:32,231 DEBUG [Thread-4014] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:54655 2023-04-28 08:56:32,231 DEBUG [Thread-4014] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:32,233 DEBUG [Thread-4018] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d765193 to 127.0.0.1:54655 2023-04-28 08:56:32,233 DEBUG [Thread-4018] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:32,269 DEBUG [Thread-4011] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-28 08:56:32,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-28 08:56:32,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672252317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672252318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672252319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672252324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,508 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/67a525e1f8404f5b85c769fc56e81527 2023-04-28 08:56:32,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59100 deadline: 1682672252820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59092 deadline: 1682672252822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59132 deadline: 1682672252823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-28 08:56:32,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.20:59060 deadline: 1682672252829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:32,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/09b85ae505dc4beaa7f012f67114c791 2023-04-28 08:56:32,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/32c2f7603686453f8de7d19507e7b4eb as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/32c2f7603686453f8de7d19507e7b4eb 2023-04-28 08:56:32,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/32c2f7603686453f8de7d19507e7b4eb, entries=350, sequenceid=678, filesize=52.9 K 2023-04-28 08:56:32,923 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/67a525e1f8404f5b85c769fc56e81527 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/67a525e1f8404f5b85c769fc56e81527 2023-04-28 08:56:32,926 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/67a525e1f8404f5b85c769fc56e81527, entries=150, sequenceid=678, filesize=11.9 K 2023-04-28 08:56:32,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/09b85ae505dc4beaa7f012f67114c791 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/09b85ae505dc4beaa7f012f67114c791 2023-04-28 08:56:32,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/09b85ae505dc4beaa7f012f67114c791, entries=150, sequenceid=678, filesize=11.9 K 2023-04-28 08:56:32,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 92c5373160b2e33e00d0bc96891cd0cb in 1281ms, sequenceid=678, compaction requested=true 2023-04-28 08:56:32,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:32,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:32,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:32,930 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:56:32,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:32,930 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:56:32,930 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107464 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:56:32,931 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:56:32,931 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/A is initiating minor compaction (all files) 2023-04-28 08:56:32,931 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/B is initiating minor compaction (all files) 2023-04-28 08:56:32,931 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/A in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:32,931 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/B in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:32,931 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f23d2651ac849a1a1e2e73fbfed386e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/7aa20d79321f40dba779421b6fd550cd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/32c2f7603686453f8de7d19507e7b4eb] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=104.9 K 2023-04-28 08:56:32,931 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5cad09c717ac4dd38c6d1ac35898fef6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/05602445be444a81a353d70df70b4d1e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/67a525e1f8404f5b85c769fc56e81527] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=36.8 K 2023-04-28 08:56:32,931 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 1f23d2651ac849a1a1e2e73fbfed386e, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1682672189459 2023-04-28 08:56:32,931 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 5cad09c717ac4dd38c6d1ac35898fef6, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1682672189459 2023-04-28 08:56:32,931 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 7aa20d79321f40dba779421b6fd550cd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1682672189491 2023-04-28 08:56:32,931 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 05602445be444a81a353d70df70b4d1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1682672189491 2023-04-28 08:56:32,932 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] compactions.Compactor(207): Compacting 32c2f7603686453f8de7d19507e7b4eb, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1682672191627 2023-04-28 08:56:32,932 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 67a525e1f8404f5b85c769fc56e81527, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1682672191643 2023-04-28 08:56:32,937 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#B#compaction#647 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:32,940 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#A#compaction#646 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:33,348 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/a7e6a59a157641c691a592cd2bb8dbb5 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a7e6a59a157641c691a592cd2bb8dbb5 2023-04-28 08:56:33,352 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/B of 92c5373160b2e33e00d0bc96891cd0cb into a7e6a59a157641c691a592cd2bb8dbb5(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:33,352 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:33,352 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/B, priority=13, startTime=1682672192930; duration=0sec 2023-04-28 08:56:33,352 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-28 08:56:33,352 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-28 08:56:33,353 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-28 08:56:33,353 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1912): 92c5373160b2e33e00d0bc96891cd0cb/C is initiating minor compaction (all files) 2023-04-28 08:56:33,353 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 92c5373160b2e33e00d0bc96891cd0cb/C in TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:33,353 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/259a161c56284054a630d9303b0ef14b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a9329d0102b1462091680a26ccf999ca, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/09b85ae505dc4beaa7f012f67114c791] into tmpdir=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp, totalSize=36.8 K 2023-04-28 08:56:33,354 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 259a161c56284054a630d9303b0ef14b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1682672189459 2023-04-28 08:56:33,354 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting a9329d0102b1462091680a26ccf999ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=665, earliestPutTs=1682672189491 2023-04-28 08:56:33,354 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] compactions.Compactor(207): Compacting 09b85ae505dc4beaa7f012f67114c791, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1682672191643 2023-04-28 08:56:33,358 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] throttle.PressureAwareThroughputController(145): 92c5373160b2e33e00d0bc96891cd0cb#C#compaction#648 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-28 08:56:33,755 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/a138552d30e04b15b1afa03573379292 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a138552d30e04b15b1afa03573379292 2023-04-28 08:56:33,758 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/A of 92c5373160b2e33e00d0bc96891cd0cb into a138552d30e04b15b1afa03573379292(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:33,758 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:33,758 INFO [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/A, priority=13, startTime=1682672192930; duration=0sec 2023-04-28 08:56:33,758 DEBUG [RS:0;jenkins-hbase6:40909-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:33,765 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/aeda743ef3744413b8b7ed35a26e8f3d as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/aeda743ef3744413b8b7ed35a26e8f3d 2023-04-28 08:56:33,768 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 92c5373160b2e33e00d0bc96891cd0cb/C of 92c5373160b2e33e00d0bc96891cd0cb into aeda743ef3744413b8b7ed35a26e8f3d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-28 08:56:33,768 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:33,768 INFO [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb., storeName=92c5373160b2e33e00d0bc96891cd0cb/C, priority=13, startTime=1682672192930; duration=0sec 2023-04-28 08:56:33,768 DEBUG [RS:0;jenkins-hbase6:40909-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-28 08:56:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40909] regionserver.HRegion(9158): Flush requested on 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:33,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-28 08:56:33,825 DEBUG [Thread-4009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08b57aae to 127.0.0.1:54655 2023-04-28 08:56:33,825 DEBUG [Thread-4009] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:33,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:33,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:33,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:33,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:33,826 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:33,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-28 08:56:33,830 DEBUG [Thread-4005] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e8654 to 127.0.0.1:54655 2023-04-28 08:56:33,830 DEBUG [Thread-4005] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:33,832 DEBUG [Thread-4003] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f06dd50 to 127.0.0.1:54655 2023-04-28 08:56:33,832 DEBUG [Thread-4003] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:33,839 DEBUG [Thread-4001] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3503b4 to 127.0.0.1:54655 2023-04-28 08:56:33,839 DEBUG [Thread-4001] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:34,237 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428ce77a805779646ddb0de5870d1c673bf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ce77a805779646ddb0de5870d1c673bf 2023-04-28 08:56:34,240 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=718, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/2c026f3925c948ec8a97eff31267455a 2023-04-28 08:56:34,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/59a7547e30f044ed99c433193322b003 2023-04-28 08:56:35,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/c58a0c2aa06f4f6280863a7aa38c9602 2023-04-28 08:56:35,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/2c026f3925c948ec8a97eff31267455a as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2c026f3925c948ec8a97eff31267455a 2023-04-28 08:56:35,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2c026f3925c948ec8a97eff31267455a, entries=150, sequenceid=718, filesize=25.4 K 2023-04-28 08:56:35,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/59a7547e30f044ed99c433193322b003 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/59a7547e30f044ed99c433193322b003 2023-04-28 08:56:35,067 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/59a7547e30f044ed99c433193322b003, entries=150, sequenceid=718, filesize=11.9 K 2023-04-28 08:56:35,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/c58a0c2aa06f4f6280863a7aa38c9602 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c58a0c2aa06f4f6280863a7aa38c9602 2023-04-28 08:56:35,070 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c58a0c2aa06f4f6280863a7aa38c9602, entries=150, sequenceid=718, filesize=11.9 K 2023-04-28 08:56:35,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=20.13 KB/20610 for 92c5373160b2e33e00d0bc96891cd0cb in 1246ms, sequenceid=718, compaction requested=false 2023-04-28 08:56:35,071 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:38,486 DEBUG [Thread-4007] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4db22237 to 127.0.0.1:54655 2023-04-28 08:56:38,486 DEBUG [Thread-4007] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:38,486 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 131 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(394): wrote 121 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 4226 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 4277 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 4102 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 4247 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(398): read 4287 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-28 08:56:38,487 INFO [Listener at localhost/39717] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 08:56:38,487 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x470742aa to 127.0.0.1:54655 2023-04-28 08:56:38,487 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:38,488 INFO [Listener at localhost/39717] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-28 08:56:38,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$11(2418): Client=jenkins//172.31.2.20 disable TestAcidGuarantees 2023-04-28 08:56:38,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:38,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 08:56:38,491 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672198491"}]},"ts":"1682672198491"} 2023-04-28 08:56:38,492 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-28 08:56:38,494 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-28 08:56:38,495 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, UNASSIGN}] 2023-04-28 08:56:38,496 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, UNASSIGN 2023-04-28 08:56:38,496 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=CLOSING, regionLocation=jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:38,497 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672198496"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682672198496"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682672198496"}]},"ts":"1682672198496"} 2023-04-28 08:56:38,497 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517}] 2023-04-28 08:56:38,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 08:56:38,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(109): Close 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 92c5373160b2e33e00d0bc96891cd0cb, disabling compactions & flushes 2023-04-28 08:56:38,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. after waiting 0 ms 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:38,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 92c5373160b2e33e00d0bc96891cd0cb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=A 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=B 2023-04-28 08:56:38,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:38,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 92c5373160b2e33e00d0bc96891cd0cb, store=C 2023-04-28 08:56:38,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-28 08:56:38,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 08:56:39,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230428e7218f074af04604abc2dd36c6de45be to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e7218f074af04604abc2dd36c6de45be 2023-04-28 08:56:39,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 08:56:39,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=725, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/304d672e0ad24d2699dbfa998874254c 2023-04-28 08:56:39,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 08:56:39,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/794cb0d8293c4a16b250c5b24c9a10e9 2023-04-28 08:56:40,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=725 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/f7c6d8d6219041e586bb7577e93ed3b9 2023-04-28 08:56:40,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/A/304d672e0ad24d2699dbfa998874254c as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/304d672e0ad24d2699dbfa998874254c 2023-04-28 08:56:40,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/304d672e0ad24d2699dbfa998874254c, entries=100, sequenceid=725, filesize=18.6 K 2023-04-28 08:56:40,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/B/794cb0d8293c4a16b250c5b24c9a10e9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/794cb0d8293c4a16b250c5b24c9a10e9 2023-04-28 08:56:40,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/794cb0d8293c4a16b250c5b24c9a10e9, entries=100, sequenceid=725, filesize=9.5 K 2023-04-28 08:56:40,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/.tmp/C/f7c6d8d6219041e586bb7577e93ed3b9 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/f7c6d8d6219041e586bb7577e93ed3b9 2023-04-28 08:56:40,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/f7c6d8d6219041e586bb7577e93ed3b9, entries=100, sequenceid=725, filesize=9.5 K 2023-04-28 08:56:40,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 92c5373160b2e33e00d0bc96891cd0cb in 1646ms, sequenceid=725, compaction requested=true 2023-04-28 08:56:40,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e9562ce0b8644a8b9092181f1788cc57, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a45f472eef0248d7b186b74a028dedb1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c2f60298b6a41eaba47871ed243eb2d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/58c6b3edec05463abfef07bf5584641f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/5ff4d96e1e684e868e9f56079f0c4383, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f22ffc198244fa0aa21649ea06dd83b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/687ad650ccd04e7b9e1a5309657c53fa, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/cba75afd865b40ada791221d889420e2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2b3d2b31e6854cabbf3eb98a5e9620a1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c296b8c3e284f8581637746304836c8, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/c73f830010c449a7ae4e941c58ac6522, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/20f0f87e917d478ab3cb146748cd92c5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ad795c18743245999ed11bdd68bf395c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a68d4e6dd8d6441b8dcf7081100439e6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/d19e26220ded4f72b3d01dc28c11acef, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ee8dafb3a40540d59de4d5d592cec36a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/33b77b37110347e1bfbe3c4dfb933784, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e2ccac1104f24150b06e4abfc90ab298, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/490eb88a536c4685b9889306c0ab607b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/8924b3f618434331a5eabb968dc2e47a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/274e7da8b24442a0aecb04ac8996c7e5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/35c04107edc94903a36df5fcc2ba8bb7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ae255786adc04c0783c0e580d656aca2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/05c1aeb7cf874558ada29435c3d0bd5c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/b5cd945b5c7b4d139b0e5edb38b657d1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9b59106d9cd3496f94e89d80495539a5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/f8d895e744ae4327a41c43687ebe3dd5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/0a4d51edfea74a6b90493d9f869cba62, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/38a4f7cca70849fb94ac5268f7a769a2, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/55fc0c0dee1f4276808a812fcf2a775b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ff4bd8f281ff480fa2903b56c8d9c5d3, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a14ddf8f3d134283a69036b957c65adc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f23d2651ac849a1a1e2e73fbfed386e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/7aa20d79321f40dba779421b6fd550cd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/32c2f7603686453f8de7d19507e7b4eb] to archive 2023-04-28 08:56:40,317 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:56:40,319 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e9562ce0b8644a8b9092181f1788cc57 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e9562ce0b8644a8b9092181f1788cc57 2023-04-28 08:56:40,320 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a45f472eef0248d7b186b74a028dedb1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a45f472eef0248d7b186b74a028dedb1 2023-04-28 08:56:40,321 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c2f60298b6a41eaba47871ed243eb2d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c2f60298b6a41eaba47871ed243eb2d 2023-04-28 08:56:40,322 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/58c6b3edec05463abfef07bf5584641f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/58c6b3edec05463abfef07bf5584641f 2023-04-28 08:56:40,323 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/5ff4d96e1e684e868e9f56079f0c4383 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/5ff4d96e1e684e868e9f56079f0c4383 2023-04-28 08:56:40,324 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f22ffc198244fa0aa21649ea06dd83b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f22ffc198244fa0aa21649ea06dd83b 2023-04-28 08:56:40,325 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/687ad650ccd04e7b9e1a5309657c53fa to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/687ad650ccd04e7b9e1a5309657c53fa 2023-04-28 08:56:40,326 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/cba75afd865b40ada791221d889420e2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/cba75afd865b40ada791221d889420e2 2023-04-28 08:56:40,327 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2b3d2b31e6854cabbf3eb98a5e9620a1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2b3d2b31e6854cabbf3eb98a5e9620a1 2023-04-28 08:56:40,328 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c296b8c3e284f8581637746304836c8 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1c296b8c3e284f8581637746304836c8 2023-04-28 08:56:40,329 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/c73f830010c449a7ae4e941c58ac6522 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/c73f830010c449a7ae4e941c58ac6522 2023-04-28 08:56:40,330 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/20f0f87e917d478ab3cb146748cd92c5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/20f0f87e917d478ab3cb146748cd92c5 2023-04-28 08:56:40,331 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ad795c18743245999ed11bdd68bf395c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ad795c18743245999ed11bdd68bf395c 2023-04-28 08:56:40,332 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a68d4e6dd8d6441b8dcf7081100439e6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a68d4e6dd8d6441b8dcf7081100439e6 2023-04-28 08:56:40,333 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/d19e26220ded4f72b3d01dc28c11acef to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/d19e26220ded4f72b3d01dc28c11acef 2023-04-28 08:56:40,334 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ee8dafb3a40540d59de4d5d592cec36a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ee8dafb3a40540d59de4d5d592cec36a 2023-04-28 08:56:40,335 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/33b77b37110347e1bfbe3c4dfb933784 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/33b77b37110347e1bfbe3c4dfb933784 2023-04-28 08:56:40,336 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e2ccac1104f24150b06e4abfc90ab298 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/e2ccac1104f24150b06e4abfc90ab298 2023-04-28 08:56:40,337 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/490eb88a536c4685b9889306c0ab607b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/490eb88a536c4685b9889306c0ab607b 2023-04-28 08:56:40,338 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/8924b3f618434331a5eabb968dc2e47a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/8924b3f618434331a5eabb968dc2e47a 2023-04-28 08:56:40,339 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/274e7da8b24442a0aecb04ac8996c7e5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/274e7da8b24442a0aecb04ac8996c7e5 2023-04-28 08:56:40,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/35c04107edc94903a36df5fcc2ba8bb7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/35c04107edc94903a36df5fcc2ba8bb7 2023-04-28 08:56:40,341 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ae255786adc04c0783c0e580d656aca2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ae255786adc04c0783c0e580d656aca2 2023-04-28 08:56:40,343 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/05c1aeb7cf874558ada29435c3d0bd5c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/05c1aeb7cf874558ada29435c3d0bd5c 2023-04-28 08:56:40,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/b5cd945b5c7b4d139b0e5edb38b657d1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/b5cd945b5c7b4d139b0e5edb38b657d1 2023-04-28 08:56:40,344 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/303e8a9f3b5a4dd9a2ffde8a8c6b94f6 2023-04-28 08:56:40,345 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9b59106d9cd3496f94e89d80495539a5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9b59106d9cd3496f94e89d80495539a5 2023-04-28 08:56:40,346 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/f8d895e744ae4327a41c43687ebe3dd5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/f8d895e744ae4327a41c43687ebe3dd5 2023-04-28 08:56:40,347 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/0a4d51edfea74a6b90493d9f869cba62 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/0a4d51edfea74a6b90493d9f869cba62 2023-04-28 08:56:40,348 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/9a2b5c0d6c924a3da87c34c2cc7f8ff1 2023-04-28 08:56:40,349 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/38a4f7cca70849fb94ac5268f7a769a2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/38a4f7cca70849fb94ac5268f7a769a2 2023-04-28 08:56:40,350 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/55fc0c0dee1f4276808a812fcf2a775b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/55fc0c0dee1f4276808a812fcf2a775b 2023-04-28 08:56:40,351 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ff4bd8f281ff480fa2903b56c8d9c5d3 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/ff4bd8f281ff480fa2903b56c8d9c5d3 2023-04-28 08:56:40,352 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a14ddf8f3d134283a69036b957c65adc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a14ddf8f3d134283a69036b957c65adc 2023-04-28 08:56:40,353 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f23d2651ac849a1a1e2e73fbfed386e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/1f23d2651ac849a1a1e2e73fbfed386e 2023-04-28 08:56:40,354 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/7aa20d79321f40dba779421b6fd550cd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/7aa20d79321f40dba779421b6fd550cd 2023-04-28 08:56:40,355 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/32c2f7603686453f8de7d19507e7b4eb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/32c2f7603686453f8de7d19507e7b4eb 2023-04-28 08:56:40,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/eee7e57bc5564e318300c22e18e63594, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/633955da885a4f15858e7a3fa8afda41, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ee388e0d93d74faa92e2b5e94707e238, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/3c657b110dd24fc6b31e637110a9edff, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/1db82857eeff4529b4e478cf39083cf4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32ca0760350b4324b6ec35b55bfe137f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0f96e79fcd3743d79363d79c779d547e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d816731bbeb5467a8bec8d349cfbb5bd, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b60a63b5da940c28c3205710a37660e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32bcf56732874b4da68ebc1c345838ac, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ba3c356e3686423da48372a84db5c2a5, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5e225f00314640aba7b704cfe2d44407, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/502ce08c06f84c17a102886998339ad4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/86cf192475394c88b526b282450c03bb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/e35156735c6b4cea9f9de4c2132b17cb, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/13b9e7713bd14b34a442107e75725b47, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/cfc092d6e3fa49a6b3d2d5816dac4e1a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2ab2ef8697c4471496855e73e68ca848, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d6af57fc34a54a2eb4257a07aebfa0e4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/41017c3eeb204159a389d7c281572a05, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/9b59721c3f2d4283bcd89b9eac5604ae, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/feb72acb408b49ac949af95cb2585755, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/bd2a53355f3e44b1ae5c6bd8a6a0521f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d8e95291f8464e2f8179c0bc0217cd02, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/fa01e53cd5fc4d06a1af4d92de9fbbc6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0da60f5ae61a43788809accc2cb077cf, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d297d5c124484865a9ae35763d15ba2a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/58c24cc9b6bd43649f8316cc3795f8f7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ed5e9ab0b5d34bc092c91b60a164a706, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/825dd9f8929b44238c882afc32527224, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/641e0fa5b97d491a91d076eea6470a2e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a0964647089c4d3abf0f091afe3d9460, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/b2346bc2852f44b682c6d06df3cc5ad4, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5cad09c717ac4dd38c6d1ac35898fef6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b6298010513465c8650dc57636e71d7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/05602445be444a81a353d70df70b4d1e, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/67a525e1f8404f5b85c769fc56e81527] to archive 2023-04-28 08:56:40,358 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:56:40,359 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/eee7e57bc5564e318300c22e18e63594 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/eee7e57bc5564e318300c22e18e63594 2023-04-28 08:56:40,360 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/633955da885a4f15858e7a3fa8afda41 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/633955da885a4f15858e7a3fa8afda41 2023-04-28 08:56:40,361 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ee388e0d93d74faa92e2b5e94707e238 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ee388e0d93d74faa92e2b5e94707e238 2023-04-28 08:56:40,362 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/3c657b110dd24fc6b31e637110a9edff to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/3c657b110dd24fc6b31e637110a9edff 2023-04-28 08:56:40,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/1db82857eeff4529b4e478cf39083cf4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/1db82857eeff4529b4e478cf39083cf4 2023-04-28 08:56:40,363 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32ca0760350b4324b6ec35b55bfe137f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32ca0760350b4324b6ec35b55bfe137f 2023-04-28 08:56:40,364 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0f96e79fcd3743d79363d79c779d547e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0f96e79fcd3743d79363d79c779d547e 2023-04-28 08:56:40,365 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d816731bbeb5467a8bec8d349cfbb5bd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d816731bbeb5467a8bec8d349cfbb5bd 2023-04-28 08:56:40,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b60a63b5da940c28c3205710a37660e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b60a63b5da940c28c3205710a37660e 2023-04-28 08:56:40,366 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32bcf56732874b4da68ebc1c345838ac to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/32bcf56732874b4da68ebc1c345838ac 2023-04-28 08:56:40,367 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ba3c356e3686423da48372a84db5c2a5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ba3c356e3686423da48372a84db5c2a5 2023-04-28 08:56:40,368 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5e225f00314640aba7b704cfe2d44407 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5e225f00314640aba7b704cfe2d44407 2023-04-28 08:56:40,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/502ce08c06f84c17a102886998339ad4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/502ce08c06f84c17a102886998339ad4 2023-04-28 08:56:40,369 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/86cf192475394c88b526b282450c03bb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/86cf192475394c88b526b282450c03bb 2023-04-28 08:56:40,370 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/e35156735c6b4cea9f9de4c2132b17cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/e35156735c6b4cea9f9de4c2132b17cb 2023-04-28 08:56:40,371 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/13b9e7713bd14b34a442107e75725b47 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/13b9e7713bd14b34a442107e75725b47 2023-04-28 08:56:40,372 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/cfc092d6e3fa49a6b3d2d5816dac4e1a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/cfc092d6e3fa49a6b3d2d5816dac4e1a 2023-04-28 08:56:40,373 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2ab2ef8697c4471496855e73e68ca848 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2ab2ef8697c4471496855e73e68ca848 2023-04-28 08:56:40,374 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d6af57fc34a54a2eb4257a07aebfa0e4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d6af57fc34a54a2eb4257a07aebfa0e4 2023-04-28 08:56:40,375 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/41017c3eeb204159a389d7c281572a05 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/41017c3eeb204159a389d7c281572a05 2023-04-28 08:56:40,376 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/9b59721c3f2d4283bcd89b9eac5604ae to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/9b59721c3f2d4283bcd89b9eac5604ae 2023-04-28 08:56:40,377 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/feb72acb408b49ac949af95cb2585755 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/feb72acb408b49ac949af95cb2585755 2023-04-28 08:56:40,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/bd2a53355f3e44b1ae5c6bd8a6a0521f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/bd2a53355f3e44b1ae5c6bd8a6a0521f 2023-04-28 08:56:40,378 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d8e95291f8464e2f8179c0bc0217cd02 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d8e95291f8464e2f8179c0bc0217cd02 2023-04-28 08:56:40,379 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/fa01e53cd5fc4d06a1af4d92de9fbbc6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/fa01e53cd5fc4d06a1af4d92de9fbbc6 2023-04-28 08:56:40,380 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0da60f5ae61a43788809accc2cb077cf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/0da60f5ae61a43788809accc2cb077cf 2023-04-28 08:56:40,381 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d297d5c124484865a9ae35763d15ba2a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/d297d5c124484865a9ae35763d15ba2a 2023-04-28 08:56:40,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/58c24cc9b6bd43649f8316cc3795f8f7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/58c24cc9b6bd43649f8316cc3795f8f7 2023-04-28 08:56:40,382 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ed5e9ab0b5d34bc092c91b60a164a706 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/ed5e9ab0b5d34bc092c91b60a164a706 2023-04-28 08:56:40,383 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/825dd9f8929b44238c882afc32527224 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/825dd9f8929b44238c882afc32527224 2023-04-28 08:56:40,384 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/641e0fa5b97d491a91d076eea6470a2e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/641e0fa5b97d491a91d076eea6470a2e 2023-04-28 08:56:40,385 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a0964647089c4d3abf0f091afe3d9460 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a0964647089c4d3abf0f091afe3d9460 2023-04-28 08:56:40,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/b2346bc2852f44b682c6d06df3cc5ad4 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/b2346bc2852f44b682c6d06df3cc5ad4 2023-04-28 08:56:40,386 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5cad09c717ac4dd38c6d1ac35898fef6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/5cad09c717ac4dd38c6d1ac35898fef6 2023-04-28 08:56:40,387 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b6298010513465c8650dc57636e71d7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/2b6298010513465c8650dc57636e71d7 2023-04-28 08:56:40,388 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/05602445be444a81a353d70df70b4d1e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/05602445be444a81a353d70df70b4d1e 2023-04-28 08:56:40,389 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/67a525e1f8404f5b85c769fc56e81527 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/67a525e1f8404f5b85c769fc56e81527 2023-04-28 08:56:40,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a07fc6e7427d4db88791253fac3d8cff, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a2e3c085cfa5403d8894c47b58bd5bfe, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/5a521c2f67564c0484e8af6ac6420c60, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/55f8d33effbb43b5b3f312f48f88ee67, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a3e062f8dc804242a2ede0939cdde2ac, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/891a3fe21dba408caf3326f7028e8964, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4370db10f288459694e875321423d884, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/653990d207e84e6081aab134b9bf4f87, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4dc55ea28e6f4d4ba52fd2836bef56bc, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/1430f9d6e224434dbab86befc84d6ea7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/db3400cacc134c81b6cf63c367277c23, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52a7cfe9d68e49bbaa50937e234ca97f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/638df471229c45c4adb03ec68611537b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ad457d54204e412a929b4911872d5eae, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/46ccd353888c48dd834d8ca4e42e9b16, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/47c02b2fe07740c882c79f9a3cdfdb7f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/671a74933ba847d0a2da3729785ac2b7, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c8d562449caa49ee86e4bb991972cb99, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ee97096e25914ade9671beeffd5c30d6, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94b6aef839644e87bec8384552b5f365, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/9f14ea4458914ca9932ae4d0be9d036a, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/6677afcb9699488ea4de20027738523f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52926550c3364b259cae7335c6ebdce0, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/62d5b414e05045e298ce1b60ce78c242, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/bef65fd57110409dafe51e41577b809d, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a84957c9ab9a4bc59f94dac1db173434, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/e116986570554efaa009fee214a51b49, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/d9e30f025f634199801d35a8547fb3f9, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8ca3eff3b8554c4282a01a3c2fe1a75f, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94ee2029008f4893b00477860f5d4284, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/2e72ec29a75f4da089d9b0dfa4c7ff1c, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/590a2073ab8c4ee7b439ae7738fd6758, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/259a161c56284054a630d9303b0ef14b, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8db62115d8f34cca85d8bd748c461408, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a9329d0102b1462091680a26ccf999ca, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/09b85ae505dc4beaa7f012f67114c791] to archive 2023-04-28 08:56:40,392 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-28 08:56:40,393 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a07fc6e7427d4db88791253fac3d8cff to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a07fc6e7427d4db88791253fac3d8cff 2023-04-28 08:56:40,394 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a2e3c085cfa5403d8894c47b58bd5bfe to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a2e3c085cfa5403d8894c47b58bd5bfe 2023-04-28 08:56:40,395 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/5a521c2f67564c0484e8af6ac6420c60 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/5a521c2f67564c0484e8af6ac6420c60 2023-04-28 08:56:40,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/55f8d33effbb43b5b3f312f48f88ee67 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/55f8d33effbb43b5b3f312f48f88ee67 2023-04-28 08:56:40,396 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a3e062f8dc804242a2ede0939cdde2ac to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a3e062f8dc804242a2ede0939cdde2ac 2023-04-28 08:56:40,397 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/891a3fe21dba408caf3326f7028e8964 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/891a3fe21dba408caf3326f7028e8964 2023-04-28 08:56:40,398 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4370db10f288459694e875321423d884 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4370db10f288459694e875321423d884 2023-04-28 08:56:40,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/653990d207e84e6081aab134b9bf4f87 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/653990d207e84e6081aab134b9bf4f87 2023-04-28 08:56:40,399 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4dc55ea28e6f4d4ba52fd2836bef56bc to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/4dc55ea28e6f4d4ba52fd2836bef56bc 2023-04-28 08:56:40,400 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/1430f9d6e224434dbab86befc84d6ea7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/1430f9d6e224434dbab86befc84d6ea7 2023-04-28 08:56:40,401 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/db3400cacc134c81b6cf63c367277c23 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/db3400cacc134c81b6cf63c367277c23 2023-04-28 08:56:40,402 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52a7cfe9d68e49bbaa50937e234ca97f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52a7cfe9d68e49bbaa50937e234ca97f 2023-04-28 08:56:40,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/638df471229c45c4adb03ec68611537b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/638df471229c45c4adb03ec68611537b 2023-04-28 08:56:40,403 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ad457d54204e412a929b4911872d5eae to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ad457d54204e412a929b4911872d5eae 2023-04-28 08:56:40,404 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/46ccd353888c48dd834d8ca4e42e9b16 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/46ccd353888c48dd834d8ca4e42e9b16 2023-04-28 08:56:40,406 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/47c02b2fe07740c882c79f9a3cdfdb7f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/47c02b2fe07740c882c79f9a3cdfdb7f 2023-04-28 08:56:40,407 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/671a74933ba847d0a2da3729785ac2b7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/671a74933ba847d0a2da3729785ac2b7 2023-04-28 08:56:40,408 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c8d562449caa49ee86e4bb991972cb99 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c8d562449caa49ee86e4bb991972cb99 2023-04-28 08:56:40,409 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ee97096e25914ade9671beeffd5c30d6 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/ee97096e25914ade9671beeffd5c30d6 2023-04-28 08:56:40,410 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94b6aef839644e87bec8384552b5f365 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94b6aef839644e87bec8384552b5f365 2023-04-28 08:56:40,411 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/9f14ea4458914ca9932ae4d0be9d036a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/9f14ea4458914ca9932ae4d0be9d036a 2023-04-28 08:56:40,412 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/6677afcb9699488ea4de20027738523f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/6677afcb9699488ea4de20027738523f 2023-04-28 08:56:40,413 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52926550c3364b259cae7335c6ebdce0 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/52926550c3364b259cae7335c6ebdce0 2023-04-28 08:56:40,414 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/62d5b414e05045e298ce1b60ce78c242 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/62d5b414e05045e298ce1b60ce78c242 2023-04-28 08:56:40,416 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/bef65fd57110409dafe51e41577b809d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/bef65fd57110409dafe51e41577b809d 2023-04-28 08:56:40,417 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a84957c9ab9a4bc59f94dac1db173434 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a84957c9ab9a4bc59f94dac1db173434 2023-04-28 08:56:40,418 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/e116986570554efaa009fee214a51b49 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/e116986570554efaa009fee214a51b49 2023-04-28 08:56:40,419 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/d9e30f025f634199801d35a8547fb3f9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/d9e30f025f634199801d35a8547fb3f9 2023-04-28 08:56:40,419 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8ca3eff3b8554c4282a01a3c2fe1a75f to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8ca3eff3b8554c4282a01a3c2fe1a75f 2023-04-28 08:56:40,420 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94ee2029008f4893b00477860f5d4284 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/94ee2029008f4893b00477860f5d4284 2023-04-28 08:56:40,421 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/2e72ec29a75f4da089d9b0dfa4c7ff1c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/2e72ec29a75f4da089d9b0dfa4c7ff1c 2023-04-28 08:56:40,422 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/590a2073ab8c4ee7b439ae7738fd6758 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/590a2073ab8c4ee7b439ae7738fd6758 2023-04-28 08:56:40,423 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/259a161c56284054a630d9303b0ef14b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/259a161c56284054a630d9303b0ef14b 2023-04-28 08:56:40,424 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8db62115d8f34cca85d8bd748c461408 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/8db62115d8f34cca85d8bd748c461408 2023-04-28 08:56:40,425 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a9329d0102b1462091680a26ccf999ca to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/a9329d0102b1462091680a26ccf999ca 2023-04-28 08:56:40,426 DEBUG [StoreCloser-TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/09b85ae505dc4beaa7f012f67114c791 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/09b85ae505dc4beaa7f012f67114c791 2023-04-28 08:56:40,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/recovered.edits/728.seqid, newMaxSeqId=728, maxSeqId=4 2023-04-28 08:56:40,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb. 2023-04-28 08:56:40,431 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 92c5373160b2e33e00d0bc96891cd0cb: 2023-04-28 08:56:40,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.UnassignRegionHandler(147): Closed 92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:40,433 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=92c5373160b2e33e00d0bc96891cd0cb, regionState=CLOSED 2023-04-28 08:56:40,433 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1682672200433"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682672200433"}]},"ts":"1682672200433"} 2023-04-28 08:56:40,434 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-28 08:56:40,435 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 92c5373160b2e33e00d0bc96891cd0cb, server=jenkins-hbase6.apache.org,40909,1682672018517 in 1.9360 sec 2023-04-28 08:56:40,436 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-28 08:56:40,436 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=92c5373160b2e33e00d0bc96891cd0cb, UNASSIGN in 1.9400 sec 2023-04-28 08:56:40,436 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682672200436"}]},"ts":"1682672200436"} 2023-04-28 08:56:40,437 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-28 08:56:40,438 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-28 08:56:40,440 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9510 sec 2023-04-28 08:56:40,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-28 08:56:40,595 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-28 08:56:40,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.HMaster$5(2228): Client=jenkins//172.31.2.20 delete TestAcidGuarantees 2023-04-28 08:56:40,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:40,598 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:40,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-28 08:56:40,598 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:40,602 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:40,604 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C, FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/recovered.edits] 2023-04-28 08:56:40,606 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2c026f3925c948ec8a97eff31267455a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/2c026f3925c948ec8a97eff31267455a 2023-04-28 08:56:40,607 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/304d672e0ad24d2699dbfa998874254c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/304d672e0ad24d2699dbfa998874254c 2023-04-28 08:56:40,608 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a138552d30e04b15b1afa03573379292 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/A/a138552d30e04b15b1afa03573379292 2023-04-28 08:56:40,610 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/59a7547e30f044ed99c433193322b003 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/59a7547e30f044ed99c433193322b003 2023-04-28 08:56:40,611 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/794cb0d8293c4a16b250c5b24c9a10e9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/794cb0d8293c4a16b250c5b24c9a10e9 2023-04-28 08:56:40,612 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a7e6a59a157641c691a592cd2bb8dbb5 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/B/a7e6a59a157641c691a592cd2bb8dbb5 2023-04-28 08:56:40,614 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/aeda743ef3744413b8b7ed35a26e8f3d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/aeda743ef3744413b8b7ed35a26e8f3d 2023-04-28 08:56:40,615 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c58a0c2aa06f4f6280863a7aa38c9602 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/c58a0c2aa06f4f6280863a7aa38c9602 2023-04-28 08:56:40,616 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/f7c6d8d6219041e586bb7577e93ed3b9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/C/f7c6d8d6219041e586bb7577e93ed3b9 2023-04-28 08:56:40,619 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/recovered.edits/728.seqid to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb/recovered.edits/728.seqid 2023-04-28 08:56:40,619 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/.tmp/data/default/TestAcidGuarantees/92c5373160b2e33e00d0bc96891cd0cb 2023-04-28 08:56:40,619 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-28 08:56:40,620 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 08:56:40,620 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-28 08:56:40,623 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428066eab1e98e34c349550828586d91a41 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428066eab1e98e34c349550828586d91a41 2023-04-28 08:56:40,624 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428074a405464344bb38189462efc801618 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428074a405464344bb38189462efc801618 2023-04-28 08:56:40,624 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280a0493fbad144787b876b1e52246e002 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304280a0493fbad144787b876b1e52246e002 2023-04-28 08:56:40,625 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281401f781505d4501833aa2eb430dafef to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281401f781505d4501833aa2eb430dafef 2023-04-28 08:56:40,626 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042815d4febd2f324d11bcc5e121903c826d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042815d4febd2f324d11bcc5e121903c826d 2023-04-28 08:56:40,627 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281617c65665254b8ab07ebf4e67dfd182 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304281617c65665254b8ab07ebf4e67dfd182 2023-04-28 08:56:40,628 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428296cfd30c20a43489e8248ebfeb22856 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428296cfd30c20a43489e8248ebfeb22856 2023-04-28 08:56:40,628 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282e42e31effc04938a14e20501f03928a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304282e42e31effc04938a14e20501f03928a 2023-04-28 08:56:40,629 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428389995f42e7840e4af999b07e6ec7539 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428389995f42e7840e4af999b07e6ec7539 2023-04-28 08:56:40,630 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283be9c008db5a4394b09c32bcde8de36d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304283be9c008db5a4394b09c32bcde8de36d 2023-04-28 08:56:40,631 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428415be84d8c7e4ebcbf5ad07b14a1247e to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428415be84d8c7e4ebcbf5ad07b14a1247e 2023-04-28 08:56:40,632 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042853d686f458e140079a249c8f30609c82 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042853d686f458e140079a249c8f30609c82 2023-04-28 08:56:40,632 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304285a0b63ce13c2423392d95348d7f7254c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304285a0b63ce13c2423392d95348d7f7254c 2023-04-28 08:56:40,633 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042870bf6bd58810416ab91fc5223c68ebf1 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042870bf6bd58810416ab91fc5223c68ebf1 2023-04-28 08:56:40,634 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428873482e652034ae897ccce013e60763c to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428873482e652034ae897ccce013e60763c 2023-04-28 08:56:40,635 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042892ecad97be714013878c0d91611052d9 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042892ecad97be714013878c0d91611052d9 2023-04-28 08:56:40,636 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428957f27849f2945df83f7439fb058b15a to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428957f27849f2945df83f7439fb058b15a 2023-04-28 08:56:40,638 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b346e541aeef4e4694c298991c1450cb to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b346e541aeef4e4694c298991c1450cb 2023-04-28 08:56:40,639 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5743756907c4da3bea76726b9f5a7d2 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b5743756907c4da3bea76726b9f5a7d2 2023-04-28 08:56:40,640 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b688db8635044d78a4853dc78768c9fd to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428b688db8635044d78a4853dc78768c9fd 2023-04-28 08:56:40,641 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c65c391a100c4bd48d2ce6619948479b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428c65c391a100c4bd48d2ce6619948479b 2023-04-28 08:56:40,642 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ce77a805779646ddb0de5870d1c673bf to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ce77a805779646ddb0de5870d1c673bf 2023-04-28 08:56:40,643 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d0892459e2ee46c899834fb55c2dac00 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d0892459e2ee46c899834fb55c2dac00 2023-04-28 08:56:40,644 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d09a98ea3534429f86cba02855390330 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d09a98ea3534429f86cba02855390330 2023-04-28 08:56:40,645 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d5396021ceee49d99750eb20bb0b45e7 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d5396021ceee49d99750eb20bb0b45e7 2023-04-28 08:56:40,646 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d74efa20e40947168ec4a418c8f9e86b to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428d74efa20e40947168ec4a418c8f9e86b 2023-04-28 08:56:40,647 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e0a91b67cb554e159237e01ddb27dd78 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e0a91b67cb554e159237e01ddb27dd78 2023-04-28 08:56:40,648 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e14f06a6a2f849d78b8e237aa0d31913 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e14f06a6a2f849d78b8e237aa0d31913 2023-04-28 08:56:40,650 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e7218f074af04604abc2dd36c6de45be to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428e7218f074af04604abc2dd36c6de45be 2023-04-28 08:56:40,651 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428eab178aacca34872a1a0353c85308e8d to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428eab178aacca34872a1a0353c85308e8d 2023-04-28 08:56:40,652 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f2cda5574cff4f1999517f7c31dd2e15 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428f2cda5574cff4f1999517f7c31dd2e15 2023-04-28 08:56:40,653 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ff508cb864dd4edab998bfa9cee1b400 to hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230428ff508cb864dd4edab998bfa9cee1b400 2023-04-28 08:56:40,653 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-28 08:56:40,655 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:40,657 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-28 08:56:40,658 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-28 08:56:40,659 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:40,659 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-28 08:56:40,659 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682672200659"}]},"ts":"9223372036854775807"} 2023-04-28 08:56:40,662 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-28 08:56:40,662 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 92c5373160b2e33e00d0bc96891cd0cb, NAME => 'TestAcidGuarantees,,1682672169138.92c5373160b2e33e00d0bc96891cd0cb.', STARTKEY => '', ENDKEY => ''}] 2023-04-28 08:56:40,662 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-28 08:56:40,663 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682672200662"}]},"ts":"9223372036854775807"} 2023-04-28 08:56:40,668 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-28 08:56:40,669 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-28 08:56:40,670 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 74 msec 2023-04-28 08:56:40,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=32801] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-28 08:56:40,700 INFO [Listener at localhost/39717] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-28 08:56:40,711 INFO [Listener at localhost/39717] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=600 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=417 (was 426), ProcessCount=170 (was 170), AvailableMemoryMB=6538 (was 6425) - AvailableMemoryMB LEAK? - 2023-04-28 08:56:40,711 INFO [Listener at localhost/39717] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-28 08:56:40,711 INFO [Listener at localhost/39717] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-28 08:56:40,711 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:54655 2023-04-28 08:56:40,711 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:40,711 DEBUG [Listener at localhost/39717] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-28 08:56:40,712 DEBUG [Listener at localhost/39717] util.JVMClusterUtil(257): Found active master hash=297978354, stopped=false 2023-04-28 08:56:40,712 INFO [Listener at localhost/39717] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:56:40,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-28 08:56:40,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-28 08:56:40,714 INFO [Listener at localhost/39717] procedure2.ProcedureExecutor(629): Stopping 2023-04-28 08:56:40,714 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:56:40,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 08:56:40,714 DEBUG [Listener at localhost/39717] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a3f12c0 to 127.0.0.1:54655 2023-04-28 08:56:40,715 DEBUG [Listener at localhost/39717] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:40,715 INFO [Listener at localhost/39717] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase6.apache.org,40909,1682672018517' ***** 2023-04-28 08:56:40,715 INFO [Listener at localhost/39717] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-28 08:56:40,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-28 08:56:40,715 INFO [RS:0;jenkins-hbase6:40909] regionserver.HeapMemoryManager(220): Stopping 2023-04-28 08:56:40,715 INFO [RS:0;jenkins-hbase6:40909] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-28 08:56:40,715 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-28 08:56:40,716 INFO [RS:0;jenkins-hbase6:40909] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-28 08:56:40,716 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(3304): Received CLOSE for a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:56:40,716 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1145): stopping server jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:40,716 DEBUG [RS:0;jenkins-hbase6:40909] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e388fd4 to 127.0.0.1:54655 2023-04-28 08:56:40,717 DEBUG [RS:0;jenkins-hbase6:40909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:40,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing a999af13cd42d26d9cb3ed05ba7639cb, disabling compactions & flushes 2023-04-28 08:56:40,717 INFO [RS:0;jenkins-hbase6:40909] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-28 08:56:40,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:56:40,717 INFO [RS:0;jenkins-hbase6:40909] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-28 08:56:40,717 INFO [RS:0;jenkins-hbase6:40909] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-28 08:56:40,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:56:40,717 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-28 08:56:40,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. after waiting 0 ms 2023-04-28 08:56:40,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:56:40,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing a999af13cd42d26d9cb3ed05ba7639cb 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-28 08:56:40,718 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-28 08:56:40,718 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, a999af13cd42d26d9cb3ed05ba7639cb=hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb.} 2023-04-28 08:56:40,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-28 08:56:40,718 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-28 08:56:40,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-28 08:56:40,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-28 08:56:40,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-28 08:56:40,718 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-28 08:56:40,720 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1505): Waiting on 1588230740, a999af13cd42d26d9cb3ed05ba7639cb 2023-04-28 08:56:40,720 INFO [regionserver/jenkins-hbase6:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-28 08:56:40,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/.tmp/info/6eddbd47150848fca88fba99004263ae 2023-04-28 08:56:40,727 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/.tmp/info/30936ac00729403497b8a7f6c23fa835 2023-04-28 08:56:40,731 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 30936ac00729403497b8a7f6c23fa835 2023-04-28 08:56:40,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/.tmp/info/6eddbd47150848fca88fba99004263ae as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/info/6eddbd47150848fca88fba99004263ae 2023-04-28 08:56:40,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/info/6eddbd47150848fca88fba99004263ae, entries=2, sequenceid=6, filesize=4.8 K 2023-04-28 08:56:40,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for a999af13cd42d26d9cb3ed05ba7639cb in 19ms, sequenceid=6, compaction requested=false 2023-04-28 08:56:40,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/namespace/a999af13cd42d26d9cb3ed05ba7639cb/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-28 08:56:40,744 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:56:40,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for a999af13cd42d26d9cb3ed05ba7639cb: 2023-04-28 08:56:40,744 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase6:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682672020524.a999af13cd42d26d9cb3ed05ba7639cb. 2023-04-28 08:56:40,901 INFO [regionserver/jenkins-hbase6:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-28 08:56:40,901 INFO [regionserver/jenkins-hbase6:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-28 08:56:40,920 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 08:56:41,120 DEBUG [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-28 08:56:41,139 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/.tmp/rep_barrier/1bfa9f4c2e1b463ba15726b4e57de912 2023-04-28 08:56:41,142 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1bfa9f4c2e1b463ba15726b4e57de912 2023-04-28 08:56:41,147 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/.tmp/table/a3863a49c174409cba215fd176cb7656 2023-04-28 08:56:41,150 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a3863a49c174409cba215fd176cb7656 2023-04-28 08:56:41,151 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/.tmp/info/30936ac00729403497b8a7f6c23fa835 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/info/30936ac00729403497b8a7f6c23fa835 2023-04-28 08:56:41,153 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 30936ac00729403497b8a7f6c23fa835 2023-04-28 08:56:41,153 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/info/30936ac00729403497b8a7f6c23fa835, entries=22, sequenceid=93, filesize=7.1 K 2023-04-28 08:56:41,154 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/.tmp/rep_barrier/1bfa9f4c2e1b463ba15726b4e57de912 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/rep_barrier/1bfa9f4c2e1b463ba15726b4e57de912 2023-04-28 08:56:41,157 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1bfa9f4c2e1b463ba15726b4e57de912 2023-04-28 08:56:41,157 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/rep_barrier/1bfa9f4c2e1b463ba15726b4e57de912, entries=6, sequenceid=93, filesize=5.5 K 2023-04-28 08:56:41,157 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/.tmp/table/a3863a49c174409cba215fd176cb7656 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/table/a3863a49c174409cba215fd176cb7656 2023-04-28 08:56:41,159 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a3863a49c174409cba215fd176cb7656 2023-04-28 08:56:41,160 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/table/a3863a49c174409cba215fd176cb7656, entries=9, sequenceid=93, filesize=5.5 K 2023-04-28 08:56:41,160 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 442ms, sequenceid=93, compaction requested=false 2023-04-28 08:56:41,167 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-28 08:56:41,167 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-28 08:56:41,168 INFO [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-28 08:56:41,168 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-28 08:56:41,168 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase6:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-28 08:56:41,320 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1171): stopping server jenkins-hbase6.apache.org,40909,1682672018517; all regions closed. 2023-04-28 08:56:41,328 DEBUG [RS:0;jenkins-hbase6:40909] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/oldWALs 2023-04-28 08:56:41,328 INFO [RS:0;jenkins-hbase6:40909] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase6.apache.org%2C40909%2C1682672018517.meta:.meta(num 1682672020224) 2023-04-28 08:56:41,332 DEBUG [RS:0;jenkins-hbase6:40909] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/oldWALs 2023-04-28 08:56:41,332 INFO [RS:0;jenkins-hbase6:40909] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase6.apache.org%2C40909%2C1682672018517:(num 1682672020023) 2023-04-28 08:56:41,332 DEBUG [RS:0;jenkins-hbase6:40909] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:41,332 INFO [RS:0;jenkins-hbase6:40909] regionserver.LeaseManager(133): Closed leases 2023-04-28 08:56:41,333 INFO [RS:0;jenkins-hbase6:40909] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase6:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-28 08:56:41,333 INFO [regionserver/jenkins-hbase6:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-28 08:56:41,334 INFO [RS:0;jenkins-hbase6:40909] ipc.NettyRpcServer(158): Stopping server on /172.31.2.20:40909 2023-04-28 08:56:41,340 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase6.apache.org,40909,1682672018517 2023-04-28 08:56:41,340 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 08:56:41,341 ERROR [Listener at localhost/39717-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5f2bd4ca rejected from java.util.concurrent.ThreadPoolExecutor@5b144bf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2558] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 08:56:41,341 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-28 08:56:41,341 ERROR [Listener at localhost/39717-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ece3dac rejected from java.util.concurrent.ThreadPoolExecutor@5b144bf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2558] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 08:56:41,342 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase6.apache.org,40909,1682672018517] 2023-04-28 08:56:41,343 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase6.apache.org,40909,1682672018517; numProcessing=1 2023-04-28 08:56:41,344 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase6.apache.org,40909,1682672018517 already deleted, retry=false 2023-04-28 08:56:41,344 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase6.apache.org,40909,1682672018517 expired; onlineServers=0 2023-04-28 08:56:41,344 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase6.apache.org,32801,1682672017275' ***** 2023-04-28 08:56:41,344 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-28 08:56:41,344 DEBUG [M:0;jenkins-hbase6:32801] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50ffc3c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase6.apache.org/172.31.2.20:0 2023-04-28 08:56:41,344 INFO [M:0;jenkins-hbase6:32801] regionserver.HRegionServer(1145): stopping server jenkins-hbase6.apache.org,32801,1682672017275 2023-04-28 08:56:41,344 INFO [M:0;jenkins-hbase6:32801] regionserver.HRegionServer(1171): stopping server jenkins-hbase6.apache.org,32801,1682672017275; all regions closed. 2023-04-28 08:56:41,344 DEBUG [M:0;jenkins-hbase6:32801] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-28 08:56:41,344 DEBUG [M:0;jenkins-hbase6:32801] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-28 08:56:41,345 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-28 08:56:41,345 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.small.0-1682672019724] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.small.0-1682672019724,5,FailOnTimeoutGroup] 2023-04-28 08:56:41,345 DEBUG [master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.large.0-1682672019723] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase6:0:becomeActiveMaster-HFileCleaner.large.0-1682672019723,5,FailOnTimeoutGroup] 2023-04-28 08:56:41,345 DEBUG [M:0;jenkins-hbase6:32801] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-28 08:56:41,345 INFO [M:0;jenkins-hbase6:32801] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-28 08:56:41,345 INFO [M:0;jenkins-hbase6:32801] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-28 08:56:41,345 INFO [M:0;jenkins-hbase6:32801] hbase.ChoreService(369): Chore service for: master/jenkins-hbase6:0 had [] on shutdown 2023-04-28 08:56:41,346 DEBUG [M:0;jenkins-hbase6:32801] master.HMaster(1512): Stopping service threads 2023-04-28 08:56:41,346 INFO [M:0;jenkins-hbase6:32801] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-28 08:56:41,346 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-28 08:56:41,346 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-28 08:56:41,346 ERROR [M:0;jenkins-hbase6:32801] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-28 08:56:41,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-28 08:56:41,347 INFO [M:0;jenkins-hbase6:32801] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-28 08:56:41,347 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-28 08:56:41,347 DEBUG [M:0;jenkins-hbase6:32801] zookeeper.ZKUtil(398): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-28 08:56:41,347 WARN [M:0;jenkins-hbase6:32801] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-28 08:56:41,347 INFO [M:0;jenkins-hbase6:32801] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-28 08:56:41,347 INFO [M:0;jenkins-hbase6:32801] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-28 08:56:41,348 DEBUG [M:0;jenkins-hbase6:32801] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-28 08:56:41,348 INFO [M:0;jenkins-hbase6:32801] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:56:41,348 DEBUG [M:0;jenkins-hbase6:32801] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:56:41,348 DEBUG [M:0;jenkins-hbase6:32801] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-28 08:56:41,348 DEBUG [M:0;jenkins-hbase6:32801] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:56:41,348 INFO [M:0;jenkins-hbase6:32801] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.90 KB heapSize=393.72 KB 2023-04-28 08:56:41,360 INFO [M:0;jenkins-hbase6:32801] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.90 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d443c611150641bab1782edc933587b1 2023-04-28 08:56:41,364 INFO [M:0;jenkins-hbase6:32801] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d443c611150641bab1782edc933587b1 2023-04-28 08:56:41,365 DEBUG [M:0;jenkins-hbase6:32801] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d443c611150641bab1782edc933587b1 as hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d443c611150641bab1782edc933587b1 2023-04-28 08:56:41,368 INFO [M:0;jenkins-hbase6:32801] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d443c611150641bab1782edc933587b1 2023-04-28 08:56:41,368 INFO [M:0;jenkins-hbase6:32801] regionserver.HStore(1080): Added hdfs://localhost:33889/user/jenkins/test-data/9efec8f0-a939-47fb-7b66-6c367cb93143/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d443c611150641bab1782edc933587b1, entries=65, sequenceid=579, filesize=24.7 K 2023-04-28 08:56:41,368 INFO [M:0;jenkins-hbase6:32801] regionserver.HRegion(2948): Finished flush of dataSize ~344.90 KB/353175, heapSize ~393.70 KB/403152, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=579, compaction requested=false 2023-04-28 08:56:41,370 INFO [M:0;jenkins-hbase6:32801] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-28 08:56:41,370 DEBUG [M:0;jenkins-hbase6:32801] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-28 08:56:41,373 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-28 08:56:41,373 INFO [M:0;jenkins-hbase6:32801] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-28 08:56:41,373 INFO [M:0;jenkins-hbase6:32801] ipc.NettyRpcServer(158): Stopping server on /172.31.2.20:32801 2023-04-28 08:56:41,376 DEBUG [M:0;jenkins-hbase6:32801] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase6.apache.org,32801,1682672017275 already deleted, retry=false 2023-04-28 08:56:41,443 INFO [RS:0;jenkins-hbase6:40909] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase6.apache.org,40909,1682672018517; zookeeper connection closed. 2023-04-28 08:56:41,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 08:56:41,443 ERROR [Listener at localhost/39717-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@56f09cc7 rejected from java.util.concurrent.ThreadPoolExecutor@5b144bf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2558] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 08:56:41,443 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@73318da6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@73318da6 2023-04-28 08:56:41,443 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): regionserver:40909-0x100f6e3c0280001, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 08:56:41,443 ERROR [Listener at localhost/39717-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@21c55088 rejected from java.util.concurrent.ThreadPoolExecutor@5b144bf7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2558] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 08:56:41,444 INFO [Listener at localhost/39717] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-28 08:56:41,543 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 08:56:41,543 INFO [M:0;jenkins-hbase6:32801] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase6.apache.org,32801,1682672017275; zookeeper connection closed. 2023-04-28 08:56:41,543 ERROR [Listener at localhost/39717-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4af1cf36 rejected from java.util.concurrent.ThreadPoolExecutor@604fe08d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4266] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 08:56:41,543 DEBUG [Listener at localhost/39717-EventThread] zookeeper.ZKWatcher(600): master:32801-0x100f6e3c0280000, quorum=127.0.0.1:54655, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-28 08:56:41,543 ERROR [Listener at localhost/39717-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20d3e23e rejected from java.util.concurrent.ThreadPoolExecutor@604fe08d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4266] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-28 08:56:41,545 WARN [Listener at localhost/39717] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-28 08:56:41,548 INFO [Listener at localhost/39717] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-28 08:56:41,656 WARN [BP-1648539919-172.31.2.20-1682672013798 heartbeating to localhost/127.0.0.1:33889] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-28 08:56:41,656 WARN [BP-1648539919-172.31.2.20-1682672013798 heartbeating to localhost/127.0.0.1:33889] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1648539919-172.31.2.20-1682672013798 (Datanode Uuid 300a562e-94bc-4fc2-8ca0-a0cb56059f0c) service to localhost/127.0.0.1:33889 2023-04-28 08:56:41,659 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/cluster_6953ce0a-bffa-891d-c013-266678f3cf8c/dfs/data/data1/current/BP-1648539919-172.31.2.20-1682672013798] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-28 08:56:41,660 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b62a084e-9302-9611-3ced-f357ad85dac3/cluster_6953ce0a-bffa-891d-c013-266678f3cf8c/dfs/data/data2/current/BP-1648539919-172.31.2.20-1682672013798] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-28 08:56:41,705 INFO [Listener at localhost/39717] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-28 08:56:41,878 INFO [Listener at localhost/39717] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-28 08:56:42,060 INFO [Listener at localhost/39717] hbase.HBaseTestingUtility(1293): Minicluster is down